2020-11-09T17:53:43.351 INFO:root:teuthology version: 1.0.0-79c79b9 2020-11-09T17:53:43.353 DEBUG:teuthology.run:Teuthology command: teuthology --block --description rgw/multisite/{clusters frontend/civetweb omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} --lock --name dang-2020-11-09_12:38:28-rgw-wip-dang-webdir-empty-object-distro-basic-smithi --verbose --owner scheduled_dang@sidious.arb.redhat.com --archive /home/teuthworker/archive/dang-2020-11-09_12:38:28-rgw-wip-dang-webdir-empty-object-distro-basic-smithi/5605822 -- /tmp/teuthology-worker.gedr80ri.tmp 2020-11-09T17:53:43.389 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-09T17:53:43.451 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/dang-2020-11-09_12:38:28-rgw-wip-dang-webdir-empty-object-distro-basic-smithi/5605822 branch: wip-dang-webdir-empty-object description: rgw/multisite/{clusters frontend/civetweb omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} email: null first_in_suite: false job_id: '5605822' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: dang-2020-11-09_12:38:28-rgw-wip-dang-webdir-empty-object-distro-basic-smithi nuke-on-error: true os_type: ubuntu overrides: admin_socket: branch: wip-dang-webdir-empty-object ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync log trim interval: 0 rgw sync obj etag verify: true setgroup: ceph setuser: ceph global: osd heartbeat grace: 40 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon osd crush smoke test: false osd: debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 osd fast shutdown: false rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 log-ignorelist: - \(PG_AVAILABILITY\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: a0560c8e5c6c33c4655e9d2a8b022793ae57419f wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: a0560c8e5c6c33c4655e9d2a8b022793ae57419f rgw: compression type: random frontend: civetweb rgw-multisite: realm: is default: true name: test-realm zonegroups: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 - endpoints: - c1.client.1 name: test-zone3 - endpoints: - c2.client.1 is_pubsub: true name: test-zone4 workunit: branch: wip-dang-webdir-empty-object sha1: a0560c8e5c6c33c4655e9d2a8b022793ae57419f owner: scheduled_dang@sidious.arb.redhat.com priority: 100 repo: https://github.com/ceph/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 sha1: a0560c8e5c6c33c4655e9d2a8b022793ae57419f sleep_before_teardown: 0 suite: rgw suite_branch: wip-dang-webdir-empty-object suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_wip-dang-webdir-empty-object/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: a0560c8e5c6c33c4655e9d2a8b022793ae57419f tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology_branch: master tube: smithi verbose: true worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.7071 2020-11-09T17:53:43.452 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_wip-dang-webdir-empty-object/qa; will attempt to use it 2020-11-09T17:53:43.453 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_wip-dang-webdir-empty-object/qa/tasks 2020-11-09T17:53:43.453 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-11-09T17:53:43.455 INFO:teuthology.task.internal:Checking packages... 2020-11-09T17:53:43.475 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'basic' and ceph hash 'a0560c8e5c6c33c4655e9d2a8b022793ae57419f' 2020-11-09T17:53:43.476 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-11-09T17:53:43.476 INFO:teuthology.packaging:ref: None 2020-11-09T17:53:43.476 INFO:teuthology.packaging:tag: None 2020-11-09T17:53:43.477 INFO:teuthology.packaging:branch: wip-dang-webdir-empty-object 2020-11-09T17:53:43.477 INFO:teuthology.packaging:sha1: a0560c8e5c6c33c4655e9d2a8b022793ae57419f 2020-11-09T17:53:43.477 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=wip-dang-webdir-empty-object 2020-11-09T17:53:43.644 INFO:teuthology.task.internal:Found packages for ceph version 16.0.0-7048-ga0560c8e-1bionic 2020-11-09T17:53:43.645 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-11-09T17:53:43.651 INFO:teuthology.task.internal:no buildpackages task found 2020-11-09T17:53:43.652 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-11-09T17:53:43.671 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-11-09T17:53:43.672 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-09T17:53:43.721 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-09T17:53:53.739 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-09T17:54:03.756 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-09T17:54:13.789 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-09T17:54:23.807 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-09T17:54:33.823 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-09T17:54:43.837 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-09T17:54:53.853 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-09T17:55:03.870 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-09T17:55:13.888 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-09T17:55:23.902 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-09T17:55:33.919 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-09T17:55:43.935 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-09T17:55:53.943 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-09T17:56:03.960 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-09T17:56:13.979 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-09T17:56:23.994 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-09T17:56:34.011 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-09T17:56:44.029 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-09T17:56:54.046 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-09T17:57:04.063 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-09T17:57:14.080 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-09T17:57:24.097 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-09T17:57:34.115 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-09T17:57:44.132 DEBUG:teuthology.lock.ops:lock_many request: {'locked_by': 'scheduled_dang@sidious.arb.redhat.com', 'count': 2, 'machine_type': 'smithi', 'description': '/home/teuthworker/archive/dang-2020-11-09_12:38:28-rgw-wip-dang-webdir-empty-object-distro-basic-smithi/5605822'} 2020-11-09T17:57:44.154 DEBUG:teuthology.lock.ops:locked ubuntu@smithi051.front.sepia.ceph.com, ubuntu@smithi078.front.sepia.ceph.com 2020-11-09T17:57:44.213 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f20d4f8ae18>, signals=[15]) 2020-11-09T17:57:44.428 INFO:teuthology.provision.fog.smithi051:Scheduling deploy of ubuntu 18.04 2020-11-09T17:57:44.432 INFO:teuthology.provision.fog.smithi078:Scheduling deploy of ubuntu 18.04 2020-11-09T17:57:44.795 INFO:teuthology.orchestra.console:Power off smithi051 2020-11-09T17:57:44.796 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi051.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-11-09T17:57:44.814 INFO:teuthology.orchestra.console:Power off smithi078 2020-11-09T17:57:44.814 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-11-09T17:57:44.824 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi051.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-09T17:57:44.837 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-09T17:57:44.947 INFO:teuthology.orchestra.console:Power off for smithi051 completed 2020-11-09T17:57:44.958 INFO:teuthology.orchestra.console:Power off for smithi078 completed 2020-11-09T17:57:45.049 INFO:teuthology.orchestra.console:Power on smithi051 2020-11-09T17:57:45.049 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi051.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-11-09T17:57:45.060 INFO:teuthology.orchestra.console:Power on smithi078 2020-11-09T17:57:45.060 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-11-09T17:57:45.071 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi051.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-09T17:57:45.085 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-09T17:57:49.102 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi051.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-09T17:57:49.112 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-09T17:57:53.214 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi051.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-09T17:57:53.224 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-09T17:57:53.435 INFO:teuthology.orchestra.console:Power on for smithi078 completed 2020-11-09T17:57:53.486 INFO:teuthology.orchestra.console:Power on for smithi051 completed 2020-11-09T17:57:53.537 INFO:teuthology.provision.fog.smithi078:Waiting for deploy to finish 2020-11-09T17:57:53.588 INFO:teuthology.provision.fog.smithi051:Waiting for deploy to finish 2020-11-09T18:00:24.962 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi051.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T18:00:49.332 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi051.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T18:00:55.178 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T18:00:58.383 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi051.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T18:01:04.243 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T18:01:07.439 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi051.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T18:01:13.299 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T18:01:16.495 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi051.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T18:01:22.354 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T18:01:31.414 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T18:01:40.503 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T18:01:49.586 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T18:02:21.129 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-11-09T18:02:22.524 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi051.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T18:02:22.610 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-11-09T18:02:22.612 INFO:teuthology.orchestra.run.smithi051:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-11-09T18:02:22.947 INFO:teuthology.provision.fog.smithi051:Node is ready 2020-11-09T18:02:22.947 INFO:teuthology.orchestra.run.smithi051:> hostname 2020-11-09T18:02:22.996 INFO:teuthology.orchestra.run.smithi051.stdout:smithi051.front.sepia.ceph.com 2020-11-09T18:02:22.997 INFO:teuthology.orchestra.run.smithi051:> grep smithi051.front.sepia.ceph.com /etc/hosts 2020-11-09T18:02:23.049 INFO:teuthology.orchestra.run.smithi051.stdout:172.21.15.51 smithi051.front.sepia.ceph.com smithi051 2020-11-09T18:02:23.049 INFO:teuthology.orchestra.run.smithi051:> sudo hostname smithi051 2020-11-09T18:02:23.140 INFO:teuthology.orchestra.run.smithi051:> sudo sed -i -e 's/smithi051.front.sepia.ceph.com/smithi051/g' /etc/hosts 2020-11-09T18:02:23.198 INFO:teuthology.orchestra.run.smithi051:> sudo sed -i -e 's/172.21.15.51/172.21.15.51/g' /etc/hosts 2020-11-09T18:02:23.253 INFO:teuthology.orchestra.run.smithi051:> sudo sed -i -e 's/smithi051.front.sepia.ceph.com/smithi051/g' /etc/hostname 2020-11-09T18:02:23.310 INFO:teuthology.orchestra.run.smithi051:> sudo hostname smithi051 2020-11-09T18:02:23.363 INFO:teuthology.provision.fog.smithi051:Deploy complete! 2020-11-09T18:02:23.363 DEBUG:teuthology.parallel:result is None 2020-11-09T18:02:27.134 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T18:02:27.144 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-11-09T18:02:33.149 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T18:02:33.160 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-11-09T18:02:39.164 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T18:02:39.174 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-11-09T18:02:45.178 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T18:02:45.189 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-11-09T18:02:51.192 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T18:02:51.271 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-11-09T18:02:51.272 INFO:teuthology.orchestra.run.smithi078:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-11-09T18:02:51.618 INFO:teuthology.provision.fog.smithi078:Node is ready 2020-11-09T18:02:51.619 INFO:teuthology.orchestra.run.smithi078:> hostname 2020-11-09T18:02:51.666 INFO:teuthology.orchestra.run.smithi078.stdout:smithi078.front.sepia.ceph.com 2020-11-09T18:02:51.667 INFO:teuthology.orchestra.run.smithi078:> grep smithi078.front.sepia.ceph.com /etc/hosts 2020-11-09T18:02:51.715 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78 smithi078.front.sepia.ceph.com smithi078 2020-11-09T18:02:51.716 INFO:teuthology.orchestra.run.smithi078:> sudo hostname smithi078 2020-11-09T18:02:51.871 INFO:teuthology.orchestra.run.smithi078:> sudo sed -i -e 's/smithi078.front.sepia.ceph.com/smithi078/g' /etc/hosts 2020-11-09T18:02:51.885 INFO:teuthology.orchestra.run.smithi078:> sudo sed -i -e 's/172.21.15.78/172.21.15.78/g' /etc/hosts 2020-11-09T18:02:51.945 INFO:teuthology.orchestra.run.smithi078:> sudo sed -i -e 's/smithi078.front.sepia.ceph.com/smithi078/g' /etc/hostname 2020-11-09T18:02:52.000 INFO:teuthology.orchestra.run.smithi078:> sudo hostname smithi078 2020-11-09T18:02:52.054 INFO:teuthology.provision.fog.smithi078:Deploy complete! 2020-11-09T18:02:52.055 DEBUG:teuthology.parallel:result is None 2020-11-09T18:02:52.212 INFO:teuthology.lock.ops:Checking smithi051.front.sepia.ceph.com 2020-11-09T18:02:52.213 INFO:teuthology.lock.ops:New key found. Updating... 2020-11-09T18:02:52.234 INFO:teuthology.lock.ops:Checking smithi078.front.sepia.ceph.com 2020-11-09T18:02:52.235 INFO:teuthology.lock.ops:New key found. Updating... 2020-11-09T18:02:52.249 INFO:teuthology.lock.ops:Updating [smithi051.front.sepia.ceph.com]: set os type and version on server 2020-11-09T18:02:52.250 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-09T18:02:52.250 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi051.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T18:02:52.328 INFO:teuthology.orchestra.run.smithi051:> true 2020-11-09T18:02:52.576 INFO:teuthology.orchestra.run.smithi051:> uname -m 2020-11-09T18:02:52.624 INFO:teuthology.orchestra.run.smithi051.stdout:x86_64 2020-11-09T18:02:52.624 INFO:teuthology.orchestra.run.smithi051:> cat /etc/os-release 2020-11-09T18:02:52.672 INFO:teuthology.orchestra.run.smithi051.stdout:NAME="Ubuntu" 2020-11-09T18:02:52.673 INFO:teuthology.orchestra.run.smithi051.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-11-09T18:02:52.673 INFO:teuthology.orchestra.run.smithi051.stdout:ID=ubuntu 2020-11-09T18:02:52.673 INFO:teuthology.orchestra.run.smithi051.stdout:ID_LIKE=debian 2020-11-09T18:02:52.674 INFO:teuthology.orchestra.run.smithi051.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-11-09T18:02:52.674 INFO:teuthology.orchestra.run.smithi051.stdout:VERSION_ID="18.04" 2020-11-09T18:02:52.674 INFO:teuthology.orchestra.run.smithi051.stdout:HOME_URL="https://www.ubuntu.com/" 2020-11-09T18:02:52.675 INFO:teuthology.orchestra.run.smithi051.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-11-09T18:02:52.675 INFO:teuthology.orchestra.run.smithi051.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-11-09T18:02:52.675 INFO:teuthology.orchestra.run.smithi051.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-11-09T18:02:52.675 INFO:teuthology.orchestra.run.smithi051.stdout:VERSION_CODENAME=bionic 2020-11-09T18:02:52.676 INFO:teuthology.orchestra.run.smithi051.stdout:UBUNTU_CODENAME=bionic 2020-11-09T18:02:52.677 INFO:teuthology.lock.ops:Updating smithi051.front.sepia.ceph.com on lock server 2020-11-09T18:02:52.695 INFO:teuthology.lock.ops:Updating [smithi078.front.sepia.ceph.com]: set os type and version on server 2020-11-09T18:02:52.695 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-09T18:02:52.696 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T18:02:52.765 INFO:teuthology.orchestra.run.smithi078:> true 2020-11-09T18:02:52.959 INFO:teuthology.orchestra.run.smithi078:> uname -m 2020-11-09T18:02:53.008 INFO:teuthology.orchestra.run.smithi078.stdout:x86_64 2020-11-09T18:02:53.009 INFO:teuthology.orchestra.run.smithi078:> cat /etc/os-release 2020-11-09T18:02:53.055 INFO:teuthology.orchestra.run.smithi078.stdout:NAME="Ubuntu" 2020-11-09T18:02:53.056 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-11-09T18:02:53.056 INFO:teuthology.orchestra.run.smithi078.stdout:ID=ubuntu 2020-11-09T18:02:53.056 INFO:teuthology.orchestra.run.smithi078.stdout:ID_LIKE=debian 2020-11-09T18:02:53.056 INFO:teuthology.orchestra.run.smithi078.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-11-09T18:02:53.056 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION_ID="18.04" 2020-11-09T18:02:53.056 INFO:teuthology.orchestra.run.smithi078.stdout:HOME_URL="https://www.ubuntu.com/" 2020-11-09T18:02:53.056 INFO:teuthology.orchestra.run.smithi078.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-11-09T18:02:53.057 INFO:teuthology.orchestra.run.smithi078.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-11-09T18:02:53.057 INFO:teuthology.orchestra.run.smithi078.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-11-09T18:02:53.057 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION_CODENAME=bionic 2020-11-09T18:02:53.057 INFO:teuthology.orchestra.run.smithi078.stdout:UBUNTU_CODENAME=bionic 2020-11-09T18:02:53.057 INFO:teuthology.lock.ops:Updating smithi078.front.sepia.ceph.com on lock server 2020-11-09T18:02:53.091 INFO:teuthology.task.internal.lock_machines:2 smithi machines locked this try, 2/2 locked so far 2020-11-09T18:02:53.125 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi051.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDChRn9ME/9o1R6oGob2yIvGJjZ3e8odesSjGsarWb9ZjpSVVq1CLSG0gjPIQwGH3lAQOoqsxc3hyAemED82m5rAOpsoeCiYN2zA5MTNVqv+oHY283y2tPAnLGcjMMIFpe/YoYBVUDoH2ppv4cpoKAD/WX31GH84Z7mXBopdTgcDZfRU6gH6YeE0/mj34rUy3Z/6oxwrNkhiMAJADRMiga/xTTAKFJQaAc4UT0JVWIp7yBdY6AVwj24vC7x35EXpJwmDwMfWAMaHxFoy/uJMazzWPyw0D7AQp857+92jl28ah1hwXTyhl4G0lqxhxX54tNsit+xwsOWwuqJpNPjUW0L smithi078.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEXAj7zwy9ounBiobhGxgHVGG0io6UH6ZgSL8GxSgCz5Xjm6d8yZtJHCZRXwzH33gb18kSLU4Pc2qngB+l7qc0UMJLU31PvCo0Nz1/x2na6haA1A0UmPtd45BOsT5v1D5taYl1dN1y26UqxxeDj0NZzpgSuF1EdSKAt75H5cBJt7mu1UlDYcn0cPnsINXRm29Bp2e4X7hyj21y7QfQL6RTa3+/8T7de4l9lhjUqV/qZRepvYWmoB9pg6U0bEHgnvkS8UQefJXYMS++teARGcP1hY3KBciTDCF+WrxHkNUNyhmRUaM3wxvILElcgkVwK4vPp+AQHJ5OmtEltNe6t/Zh 2020-11-09T18:02:53.126 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-09T18:02:53.174 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-11-09T18:02:53.186 INFO:teuthology.task.internal:Saving configuration 2020-11-09T18:02:53.211 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-11-09T18:02:53.220 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-11-09T18:02:53.239 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-11-09 17:57:44.140098', 'description': '/home/teuthworker/archive/dang-2020-11-09_12:38:28-rgw-wip-dang-webdir-empty-object-distro-basic-smithi/5605822', 'locked_by': 'scheduled_dang@sidious.arb.redhat.com', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAzPfBIgT4ik0A6Sq7sS+vj8YEmkz5EiBlQs0PON3dIl', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi051.front.sepia.ceph.com'} 2020-11-09T18:02:53.255 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-11-09 17:57:44.140119', 'description': '/home/teuthworker/archive/dang-2020-11-09_12:38:28-rgw-wip-dang-webdir-empty-object-distro-basic-smithi/5605822', 'locked_by': 'scheduled_dang@sidious.arb.redhat.com', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAMtWdx5b1uV7wJhQ8Olz8zQpgEAJbKZHUm2QZtRQK9r', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi078.front.sepia.ceph.com'} 2020-11-09T18:02:53.256 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-11-09T18:02:53.266 INFO:teuthology.task.internal:roles: ubuntu@smithi051.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2020-11-09T18:02:53.267 INFO:teuthology.task.internal:roles: ubuntu@smithi078.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2020-11-09T18:02:53.267 INFO:teuthology.run_tasks:Running task console_log... 2020-11-09T18:02:53.384 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f20d4fe9f28>, signals=[15]) 2020-11-09T18:02:53.385 INFO:teuthology.run_tasks:Running task internal.connect... 2020-11-09T18:02:53.397 INFO:teuthology.task.internal:Opening connections... 2020-11-09T18:02:53.397 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi051.front.sepia.ceph.com 2020-11-09T18:02:53.399 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi051.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T18:02:53.475 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi078.front.sepia.ceph.com 2020-11-09T18:02:53.477 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T18:02:53.552 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-11-09T18:02:53.568 INFO:teuthology.orchestra.run.smithi051:> uname -m 2020-11-09T18:02:53.682 INFO:teuthology.orchestra.run.smithi051.stdout:x86_64 2020-11-09T18:02:53.682 INFO:teuthology.orchestra.run.smithi051:> cat /etc/os-release 2020-11-09T18:02:53.727 INFO:teuthology.orchestra.run.smithi051.stdout:NAME="Ubuntu" 2020-11-09T18:02:53.728 INFO:teuthology.orchestra.run.smithi051.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-11-09T18:02:53.728 INFO:teuthology.orchestra.run.smithi051.stdout:ID=ubuntu 2020-11-09T18:02:53.728 INFO:teuthology.orchestra.run.smithi051.stdout:ID_LIKE=debian 2020-11-09T18:02:53.728 INFO:teuthology.orchestra.run.smithi051.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-11-09T18:02:53.729 INFO:teuthology.orchestra.run.smithi051.stdout:VERSION_ID="18.04" 2020-11-09T18:02:53.729 INFO:teuthology.orchestra.run.smithi051.stdout:HOME_URL="https://www.ubuntu.com/" 2020-11-09T18:02:53.729 INFO:teuthology.orchestra.run.smithi051.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-11-09T18:02:53.729 INFO:teuthology.orchestra.run.smithi051.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-11-09T18:02:53.729 INFO:teuthology.orchestra.run.smithi051.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-11-09T18:02:53.729 INFO:teuthology.orchestra.run.smithi051.stdout:VERSION_CODENAME=bionic 2020-11-09T18:02:53.730 INFO:teuthology.orchestra.run.smithi051.stdout:UBUNTU_CODENAME=bionic 2020-11-09T18:02:53.730 INFO:teuthology.lock.ops:Updating smithi051.front.sepia.ceph.com on lock server 2020-11-09T18:02:53.751 INFO:teuthology.orchestra.run.smithi078:> uname -m 2020-11-09T18:02:53.756 INFO:teuthology.orchestra.run.smithi078.stdout:x86_64 2020-11-09T18:02:53.757 INFO:teuthology.orchestra.run.smithi078:> cat /etc/os-release 2020-11-09T18:02:53.803 INFO:teuthology.orchestra.run.smithi078.stdout:NAME="Ubuntu" 2020-11-09T18:02:53.804 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-11-09T18:02:53.804 INFO:teuthology.orchestra.run.smithi078.stdout:ID=ubuntu 2020-11-09T18:02:53.804 INFO:teuthology.orchestra.run.smithi078.stdout:ID_LIKE=debian 2020-11-09T18:02:53.804 INFO:teuthology.orchestra.run.smithi078.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-11-09T18:02:53.804 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION_ID="18.04" 2020-11-09T18:02:53.804 INFO:teuthology.orchestra.run.smithi078.stdout:HOME_URL="https://www.ubuntu.com/" 2020-11-09T18:02:53.805 INFO:teuthology.orchestra.run.smithi078.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-11-09T18:02:53.805 INFO:teuthology.orchestra.run.smithi078.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-11-09T18:02:53.805 INFO:teuthology.orchestra.run.smithi078.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-11-09T18:02:53.805 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION_CODENAME=bionic 2020-11-09T18:02:53.805 INFO:teuthology.orchestra.run.smithi078.stdout:UBUNTU_CODENAME=bionic 2020-11-09T18:02:53.806 INFO:teuthology.lock.ops:Updating smithi078.front.sepia.ceph.com on lock server 2020-11-09T18:02:53.821 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-11-09T18:02:53.863 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2020-11-09T18:02:53.874 INFO:teuthology.task.internal:Checking for old test directory... 2020-11-09T18:02:53.874 INFO:teuthology.orchestra.run.smithi051:> test '!' -e /home/ubuntu/cephtest 2020-11-09T18:02:53.877 INFO:teuthology.orchestra.run.smithi078:> test '!' -e /home/ubuntu/cephtest 2020-11-09T18:02:53.882 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2020-11-09T18:02:53.924 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2020-11-09T18:02:53.924 INFO:teuthology.orchestra.run.smithi051:> test -z $(ls -A /var/lib/ceph) 2020-11-09T18:02:53.926 INFO:teuthology.orchestra.run.smithi078:> test -z $(ls -A /var/lib/ceph) 2020-11-09T18:02:53.943 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2020-11-09T18:02:53.999 INFO:teuthology.run_tasks:Running task kernel... 2020-11-09T18:02:54.043 DEBUG:teuthology.task.kernel:normalized config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c1.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c1.client.0': {'kdb': True, 'sha1': 'distro'}, 'c1.client.1': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c2.client.0': {'kdb': True, 'sha1': 'distro'}, 'c2.client.1': {'kdb': True, 'sha1': 'distro'}} 2020-11-09T18:02:54.043 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2020-11-09T18:02:54.044 INFO:teuthology.orchestra.run.smithi051:> uname -r 2020-11-09T18:02:54.050 INFO:teuthology.orchestra.run.smithi051.stdout:4.15.0-122-generic 2020-11-09T18:02:54.051 INFO:teuthology.task.kernel:Running kernel on smithi051: 4.15.0-122-generic 2020-11-09T18:02:54.051 INFO:teuthology.orchestra.run.smithi051:> sudo apt-get clean 2020-11-09T18:02:54.201 INFO:teuthology.orchestra.run.smithi051:> sudo apt-get update 2020-11-09T18:02:54.434 INFO:teuthology.orchestra.run.smithi051.stdout:Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2020-11-09T18:02:54.560 INFO:teuthology.orchestra.run.smithi051.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-11-09T18:02:54.641 INFO:teuthology.orchestra.run.smithi051.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2020-11-09T18:02:54.683 INFO:teuthology.orchestra.run.smithi051.stdout:Get:4 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [847 kB] 2020-11-09T18:02:54.771 INFO:teuthology.orchestra.run.smithi051.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,400 kB] 2020-11-09T18:02:54.796 INFO:teuthology.orchestra.run.smithi051.stdout:Get:6 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [965 kB] 2020-11-09T18:02:54.807 INFO:teuthology.orchestra.run.smithi051.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,079 kB] 2020-11-09T18:02:54.951 INFO:teuthology.orchestra.run.smithi051.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2020-11-09T18:02:55.078 INFO:teuthology.orchestra.run.smithi051.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,142 kB] 2020-11-09T18:02:55.411 INFO:teuthology.orchestra.run.smithi051.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,727 kB] 2020-11-09T18:02:55.504 INFO:teuthology.orchestra.run.smithi051.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,545 kB] 2020-11-09T18:02:55.542 INFO:teuthology.orchestra.run.smithi051.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,684 kB] 2020-11-09T18:02:56.245 INFO:teuthology.orchestra.run.smithi051.stdout:Fetched 10.6 MB in 2s (6,134 kB/s) 2020-11-09T18:02:57.743 INFO:teuthology.orchestra.run.smithi051.stdout:Reading package lists... 2020-11-09T18:02:57.792 INFO:teuthology.orchestra.run.smithi051:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-11-09T18:02:57.877 INFO:teuthology.orchestra.run.smithi051.stdout:Reading package lists... 2020-11-09T18:02:58.140 INFO:teuthology.orchestra.run.smithi051.stdout:Building dependency tree... 2020-11-09T18:02:58.141 INFO:teuthology.orchestra.run.smithi051.stdout:Reading state information... 2020-11-09T18:02:58.230 INFO:teuthology.orchestra.run.smithi051.stderr:E: Unable to locate package linux-image-current-generic 2020-11-09T18:02:58.231 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-11-09T18:02:58.232 INFO:teuthology.orchestra.run.smithi051:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-11-09T18:02:58.310 INFO:teuthology.orchestra.run.smithi051.stdout:Reading package lists... 2020-11-09T18:02:58.570 INFO:teuthology.orchestra.run.smithi051.stdout:Building dependency tree... 2020-11-09T18:02:58.571 INFO:teuthology.orchestra.run.smithi051.stdout:Reading state information... 2020-11-09T18:02:58.785 INFO:teuthology.orchestra.run.smithi051.stdout:linux-image-generic is already the newest version (4.15.0.122.109). 2020-11-09T18:02:58.785 INFO:teuthology.orchestra.run.smithi051.stdout:linux-image-generic set to manually installed. 2020-11-09T18:02:58.785 INFO:teuthology.orchestra.run.smithi051.stdout:0 upgraded, 0 newly installed, 0 to remove and 2 not upgraded. 2020-11-09T18:02:58.786 INFO:teuthology.orchestra.run.smithi051:> dpkg -s linux-image-generic 2020-11-09T18:02:58.803 INFO:teuthology.orchestra.run.smithi051.stdout:Package: linux-image-generic 2020-11-09T18:02:58.803 INFO:teuthology.orchestra.run.smithi051.stdout:Status: install ok installed 2020-11-09T18:02:58.803 INFO:teuthology.orchestra.run.smithi051.stdout:Priority: optional 2020-11-09T18:02:58.803 INFO:teuthology.orchestra.run.smithi051.stdout:Section: kernel 2020-11-09T18:02:58.804 INFO:teuthology.orchestra.run.smithi051.stdout:Installed-Size: 16 2020-11-09T18:02:58.804 INFO:teuthology.orchestra.run.smithi051.stdout:Maintainer: Ubuntu Kernel Team 2020-11-09T18:02:58.804 INFO:teuthology.orchestra.run.smithi051.stdout:Architecture: amd64 2020-11-09T18:02:58.804 INFO:teuthology.orchestra.run.smithi051.stdout:Source: linux-meta 2020-11-09T18:02:58.804 INFO:teuthology.orchestra.run.smithi051.stdout:Version: 4.15.0.122.109 2020-11-09T18:02:58.804 INFO:teuthology.orchestra.run.smithi051.stdout:Provides: wireguard-modules (= 1.0.20200611-1ubuntu1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2020-11-09T18:02:58.805 INFO:teuthology.orchestra.run.smithi051.stdout:Depends: linux-image-4.15.0-122-generic, linux-modules-extra-4.15.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2020-11-09T18:02:58.805 INFO:teuthology.orchestra.run.smithi051.stdout:Recommends: thermald 2020-11-09T18:02:58.805 INFO:teuthology.orchestra.run.smithi051.stdout:Description: Generic Linux kernel image 2020-11-09T18:02:58.805 INFO:teuthology.orchestra.run.smithi051.stdout: This package will always depend on the latest generic kernel image 2020-11-09T18:02:58.806 INFO:teuthology.orchestra.run.smithi051.stdout: available. 2020-11-09T18:02:58.806 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2020-11-09T18:02:58.806 INFO:teuthology.orchestra.run.smithi078:> uname -r 2020-11-09T18:02:58.812 INFO:teuthology.orchestra.run.smithi078.stdout:4.15.0-122-generic 2020-11-09T18:02:58.812 INFO:teuthology.task.kernel:Running kernel on smithi078: 4.15.0-122-generic 2020-11-09T18:02:58.813 INFO:teuthology.orchestra.run.smithi078:> sudo apt-get clean 2020-11-09T18:02:58.934 INFO:teuthology.orchestra.run.smithi078:> sudo apt-get update 2020-11-09T18:02:59.285 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-11-09T18:02:59.285 INFO:teuthology.orchestra.run.smithi078.stdout:Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2020-11-09T18:02:59.340 INFO:teuthology.orchestra.run.smithi078.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2020-11-09T18:02:59.638 INFO:teuthology.orchestra.run.smithi078.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2020-11-09T18:02:59.676 INFO:teuthology.orchestra.run.smithi078.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,400 kB] 2020-11-09T18:02:59.788 INFO:teuthology.orchestra.run.smithi078.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,727 kB] 2020-11-09T18:03:00.100 INFO:teuthology.orchestra.run.smithi078.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [847 kB] 2020-11-09T18:03:00.135 INFO:teuthology.orchestra.run.smithi078.stdout:Get:8 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,079 kB] 2020-11-09T18:03:00.179 INFO:teuthology.orchestra.run.smithi078.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,142 kB] 2020-11-09T18:03:00.199 INFO:teuthology.orchestra.run.smithi078.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [965 kB] 2020-11-09T18:03:00.213 INFO:teuthology.orchestra.run.smithi078.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,545 kB] 2020-11-09T18:03:00.282 INFO:teuthology.orchestra.run.smithi078.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,684 kB] 2020-11-09T18:03:01.398 INFO:teuthology.orchestra.run.smithi078.stdout:Fetched 10.6 MB in 2s (4,908 kB/s) 2020-11-09T18:03:02.865 INFO:teuthology.orchestra.run.smithi078.stdout:Reading package lists... 2020-11-09T18:03:02.911 INFO:teuthology.orchestra.run.smithi078:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-11-09T18:03:02.985 INFO:teuthology.orchestra.run.smithi078.stdout:Reading package lists... 2020-11-09T18:03:03.212 INFO:teuthology.orchestra.run.smithi078.stdout:Building dependency tree... 2020-11-09T18:03:03.213 INFO:teuthology.orchestra.run.smithi078.stdout:Reading state information... 2020-11-09T18:03:03.290 INFO:teuthology.orchestra.run.smithi078.stderr:E: Unable to locate package linux-image-current-generic 2020-11-09T18:03:03.291 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-11-09T18:03:03.292 INFO:teuthology.orchestra.run.smithi078:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-11-09T18:03:03.371 INFO:teuthology.orchestra.run.smithi078.stdout:Reading package lists... 2020-11-09T18:03:03.609 INFO:teuthology.orchestra.run.smithi078.stdout:Building dependency tree... 2020-11-09T18:03:03.609 INFO:teuthology.orchestra.run.smithi078.stdout:Reading state information... 2020-11-09T18:03:03.822 INFO:teuthology.orchestra.run.smithi078.stdout:linux-image-generic is already the newest version (4.15.0.122.109). 2020-11-09T18:03:03.822 INFO:teuthology.orchestra.run.smithi078.stdout:linux-image-generic set to manually installed. 2020-11-09T18:03:03.822 INFO:teuthology.orchestra.run.smithi078.stdout:0 upgraded, 0 newly installed, 0 to remove and 2 not upgraded. 2020-11-09T18:03:03.824 INFO:teuthology.orchestra.run.smithi078:> dpkg -s linux-image-generic 2020-11-09T18:03:03.849 INFO:teuthology.orchestra.run.smithi078.stdout:Package: linux-image-generic 2020-11-09T18:03:03.849 INFO:teuthology.orchestra.run.smithi078.stdout:Status: install ok installed 2020-11-09T18:03:03.849 INFO:teuthology.orchestra.run.smithi078.stdout:Priority: optional 2020-11-09T18:03:03.849 INFO:teuthology.orchestra.run.smithi078.stdout:Section: kernel 2020-11-09T18:03:03.849 INFO:teuthology.orchestra.run.smithi078.stdout:Installed-Size: 16 2020-11-09T18:03:03.850 INFO:teuthology.orchestra.run.smithi078.stdout:Maintainer: Ubuntu Kernel Team 2020-11-09T18:03:03.850 INFO:teuthology.orchestra.run.smithi078.stdout:Architecture: amd64 2020-11-09T18:03:03.850 INFO:teuthology.orchestra.run.smithi078.stdout:Source: linux-meta 2020-11-09T18:03:03.850 INFO:teuthology.orchestra.run.smithi078.stdout:Version: 4.15.0.122.109 2020-11-09T18:03:03.850 INFO:teuthology.orchestra.run.smithi078.stdout:Provides: wireguard-modules (= 1.0.20200611-1ubuntu1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2020-11-09T18:03:03.850 INFO:teuthology.orchestra.run.smithi078.stdout:Depends: linux-image-4.15.0-122-generic, linux-modules-extra-4.15.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2020-11-09T18:03:03.850 INFO:teuthology.orchestra.run.smithi078.stdout:Recommends: thermald 2020-11-09T18:03:03.851 INFO:teuthology.orchestra.run.smithi078.stdout:Description: Generic Linux kernel image 2020-11-09T18:03:03.851 INFO:teuthology.orchestra.run.smithi078.stdout: This package will always depend on the latest generic kernel image 2020-11-09T18:03:03.851 INFO:teuthology.orchestra.run.smithi078.stdout: available. 2020-11-09T18:03:03.851 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2020-11-09T18:03:03.851 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2020-11-09T18:03:03.852 INFO:teuthology.orchestra.run.smithi051:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-11-09T18:03:03.869 INFO:teuthology.orchestra.run.smithi051.stdout:ttyS1 2020-11-09T18:03:03.893 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2020-11-09T18:03:03.893 INFO:teuthology.orchestra.run.smithi078:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-11-09T18:03:03.907 INFO:teuthology.orchestra.run.smithi078.stdout:ttyS1 2020-11-09T18:03:03.924 INFO:teuthology.run_tasks:Running task internal.base... 2020-11-09T18:03:03.932 INFO:teuthology.task.internal:Creating test directory... 2020-11-09T18:03:03.933 INFO:teuthology.orchestra.run.smithi051:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-11-09T18:03:03.936 INFO:teuthology.orchestra.run.smithi078:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-11-09T18:03:03.970 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2020-11-09T18:03:03.981 INFO:teuthology.run_tasks:Running task internal.archive... 2020-11-09T18:03:03.994 INFO:teuthology.task.internal:Creating archive directory... 2020-11-09T18:03:03.995 INFO:teuthology.orchestra.run.smithi051:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-11-09T18:03:03.997 INFO:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-11-09T18:03:04.023 INFO:teuthology.run_tasks:Running task internal.coredump... 2020-11-09T18:03:04.034 INFO:teuthology.task.internal:Enabling coredump saving... 2020-11-09T18:03:04.034 INFO:teuthology.orchestra.run.smithi051:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-11-09T18:03:04.056 INFO:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-11-09T18:03:04.078 INFO:teuthology.orchestra.run.smithi051.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-09T18:03:04.084 INFO:teuthology.orchestra.run.smithi051.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-09T18:03:04.085 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-09T18:03:04.091 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-09T18:03:04.092 INFO:teuthology.run_tasks:Running task internal.sudo... 2020-11-09T18:03:04.102 INFO:teuthology.task.internal:Configuring sudo... 2020-11-09T18:03:04.102 INFO:teuthology.orchestra.run.smithi051:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-11-09T18:03:04.128 INFO:teuthology.orchestra.run.smithi078:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-11-09T18:03:04.145 INFO:teuthology.run_tasks:Running task internal.syslog... 2020-11-09T18:03:04.157 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2020-11-09T18:03:04.157 INFO:teuthology.orchestra.run.smithi051:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-11-09T18:03:04.184 INFO:teuthology.orchestra.run.smithi078:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-11-09T18:03:04.193 INFO:teuthology.orchestra.run.smithi051:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-11-09T18:03:04.236 INFO:teuthology.orchestra.run.smithi051:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-11-09T18:03:04.284 INFO:teuthology.orchestra.run.smithi051:> set -ex 2020-11-09T18:03:04.285 INFO:teuthology.orchestra.run.smithi051:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-09T18:03:04.330 INFO:teuthology.orchestra.run.smithi051.stderr:+ sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-09T18:03:04.342 INFO:teuthology.orchestra.run.smithi051.stderr:0+1 records in 2020-11-09T18:03:04.342 INFO:teuthology.orchestra.run.smithi051.stderr:0+1 records out 2020-11-09T18:03:04.342 INFO:teuthology.orchestra.run.smithi051.stderr:152 bytes copied, 3.1993e-05 s, 4.8 MB/s 2020-11-09T18:03:04.343 INFO:teuthology.orchestra.run.smithi078:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-11-09T18:03:04.350 INFO:teuthology.orchestra.run.smithi078:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-11-09T18:03:04.397 INFO:teuthology.orchestra.run.smithi078:> set -ex 2020-11-09T18:03:04.398 INFO:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-09T18:03:04.444 INFO:teuthology.orchestra.run.smithi078.stderr:+ sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-09T18:03:04.462 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records in 2020-11-09T18:03:04.462 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records out 2020-11-09T18:03:04.462 INFO:teuthology.orchestra.run.smithi078.stderr:152 bytes copied, 2.8801e-05 s, 5.3 MB/s 2020-11-09T18:03:04.463 INFO:teuthology.orchestra.run.smithi051:> sudo service rsyslog restart 2020-11-09T18:03:04.465 INFO:teuthology.orchestra.run.smithi078:> sudo service rsyslog restart 2020-11-09T18:03:04.547 INFO:teuthology.run_tasks:Running task internal.timer... 2020-11-09T18:03:04.556 INFO:teuthology.task.internal:Starting timer... 2020-11-09T18:03:04.557 INFO:teuthology.run_tasks:Running task pcp... 2020-11-09T18:03:04.702 INFO:teuthology.run_tasks:Running task selinux... 2020-11-09T18:03:04.787 DEBUG:teuthology.task.selinux:Excluding smithi051: OS 'ubuntu' does not support SELinux 2020-11-09T18:03:04.787 DEBUG:teuthology.task.selinux:Excluding smithi078: OS 'ubuntu' does not support SELinux 2020-11-09T18:03:04.788 DEBUG:teuthology.task.selinux:Getting current SELinux state 2020-11-09T18:03:04.788 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2020-11-09T18:03:04.788 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2020-11-09T18:03:04.788 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2020-11-09T18:03:04.826 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2020-11-09T18:03:04.833 INFO:teuthology.repo_utils:Fetching master from origin 2020-11-09T18:03:04.880 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch origin/master 2020-11-09T18:03:04.893 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2020-11-09T18:03:04.894 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi051.front.sepia.ceph.com,smithi078.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2020-11-09T18:03:05.226 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2020-11-09T18:03:06.085 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2020-11-09T18:03:08.322 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2020-11-09T18:03:08.341 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:08 +0000 (0:00:00.067) 0:00:00.067 ******* 2020-11-09T18:03:08.358 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:08 +0000 (0:00:00.016) 0:00:00.084 ******* 2020-11-09T18:03:09.895 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi078.front.sepia.ceph.com] 2020-11-09T18:03:10.065 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:10 +0000 (0:00:01.706) 0:00:01.791 ******* 2020-11-09T18:03:10.075 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] 2020-11-09T18:03:10.147 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:10 +0000 (0:00:00.082) 0:00:01.873 ******* 2020-11-09T18:03:10.362 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2020-11-09T18:03:10.364 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-11-09T18:03:10.403 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:10 +0000 (0:00:00.255) 0:00:02.129 ******* 2020-11-09T18:03:10.443 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:10 +0000 (0:00:00.039) 0:00:02.169 ******* 2020-11-09T18:03:10.868 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2020-11-09T18:03:10.870 INFO:teuthology.task.ansible.out:changed: [smithi051.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} changed: [smithi078.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-11-09T18:03:10.904 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:10 +0000 (0:00:00.461) 0:00:02.630 ******* 2020-11-09T18:03:10.940 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:10 +0000 (0:00:00.035) 0:00:02.666 ******* 2020-11-09T18:03:11.219 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2020-11-09T18:03:11.220 INFO:teuthology.task.ansible.out:changed: [smithi051.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.055750", "end": "2020-11-09 18:03:10.862592", "rc": 0, "start": "2020-11-09 18:03:10.806842", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2020-11-09T18:03:11.256 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:11 +0000 (0:00:00.316) 0:00:02.982 ******* 2020-11-09T18:03:11.264 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.068174", "end": "2020-11-09 18:03:11.679819", "rc": 0, "start": "2020-11-09 18:03:11.611645", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2020-11-09T18:03:11.304 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:11 +0000 (0:00:00.047) 0:00:03.030 ******* 2020-11-09T18:03:11.499 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2020-11-09T18:03:11.501 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} ok: [smithi051.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-09T18:03:11.543 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:11 +0000 (0:00:00.239) 0:00:03.269 ******* 2020-11-09T18:03:11.576 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:11 +0000 (0:00:00.032) 0:00:03.301 ******* 2020-11-09T18:03:12.192 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2020-11-09T18:03:12.194 INFO:teuthology.task.ansible.out:changed: [smithi051.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1604944991.5570152-182640113056355/source", "state": "file", "uid": 0} 2020-11-09T18:03:12.233 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:12 +0000 (0:00:00.657) 0:00:03.959 ******* 2020-11-09T18:03:12.240 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1604944991.5907426-129809329250384/source", "state": "file", "uid": 0} 2020-11-09T18:03:12.275 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:12 +0000 (0:00:00.041) 0:00:04.001 ******* 2020-11-09T18:03:12.537 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2020-11-09T18:03:12.539 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2020-11-09T18:03:12.541 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-11-09T18:03:12.572 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2020-11-09T18:03:12.574 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-11-09T18:03:12.691 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2020-11-09T18:03:12.697 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2020-11-09T18:03:12.830 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:12 +0000 (0:00:00.555) 0:00:04.556 ******* 2020-11-09T18:03:12.863 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:12 +0000 (0:00:00.032) 0:00:04.588 ******* 2020-11-09T18:03:13.539 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi078.front.sepia.ceph.com] 2020-11-09T18:03:13.614 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2020-11-09T18:03:13.616 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:13 +0000 (0:00:00.751) 0:00:05.340 ******* 2020-11-09T18:03:13.991 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2020-11-09T18:03:13.992 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.185327", "end": "2020-11-09 18:03:13.963364", "rc": 0, "start": "2020-11-09 18:03:13.778037", "stderr": "", "stderr_lines": [], "stdout": "3af9be36a63516879e86faffe49764ecad425070", "stdout_lines": ["3af9be36a63516879e86faffe49764ecad425070"]} 2020-11-09T18:03:14.025 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:14 +0000 (0:00:00.410) 0:00:05.751 ******* 2020-11-09T18:03:14.165 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2020-11-09T18:03:14.167 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.009905", "end": "2020-11-09 18:03:14.603743", "failed_when_result": false, "rc": 0, "start": "2020-11-09 18:03:14.593838", "stderr": "", "stderr_lines": [], "stdout": "f76e9e239faceca753ffd443c97bd3654ebef2d1", "stdout_lines": ["f76e9e239faceca753ffd443c97bd3654ebef2d1"]} 2020-11-09T18:03:14.200 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:14 +0000 (0:00:00.174) 0:00:05.925 ******* 2020-11-09T18:03:14.227 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:14.261 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:14 +0000 (0:00:00.061) 0:00:05.987 ******* 2020-11-09T18:03:14.288 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2020-11-09T18:03:14.290 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:14.322 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:14 +0000 (0:00:00.060) 0:00:06.048 ******* 2020-11-09T18:03:14.350 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2020-11-09T18:03:14.352 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:14.384 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:14 +0000 (0:00:00.062) 0:00:06.110 ******* 2020-11-09T18:03:14.417 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2020-11-09T18:03:14.418 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:14.452 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:14 +0000 (0:00:00.068) 0:00:06.178 ******* 2020-11-09T18:03:14.479 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2020-11-09T18:03:14.481 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:14.513 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:14 +0000 (0:00:00.060) 0:00:06.239 ******* 2020-11-09T18:03:14.597 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi051.front.sepia.ceph.com] 2020-11-09T18:03:14.672 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:14 +0000 (0:00:00.158) 0:00:06.397 ******* 2020-11-09T18:03:14.678 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2020-11-09T18:03:14.680 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-09T18:03:14.714 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:14 +0000 (0:00:00.041) 0:00:06.439 ******* 2020-11-09T18:03:14.900 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-11-09T18:03:14.902 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2020-11-09T18:03:15.008 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2020-11-09T18:03:15.010 INFO:teuthology.task.ansible.out:changed: [smithi051.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.152956", "end": "2020-11-09 18:03:14.986308", "rc": 0, "start": "2020-11-09 18:03:14.833352", "stderr": "", "stderr_lines": [], "stdout": "3af9be36a63516879e86faffe49764ecad425070", "stdout_lines": ["3af9be36a63516879e86faffe49764ecad425070"]} 2020-11-09T18:03:15.043 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:15 +0000 (0:00:00.330) 0:00:06.769 ******* 2020-11-09T18:03:15.050 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-11-09T18:03:15.052 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2020-11-09T18:03:15.170 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2020-11-09T18:03:15.172 INFO:teuthology.task.ansible.out:changed: [smithi051.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.004552", "end": "2020-11-09 18:03:14.822072", "failed_when_result": false, "rc": 0, "start": "2020-11-09 18:03:14.817520", "stderr": "", "stderr_lines": [], "stdout": "f76e9e239faceca753ffd443c97bd3654ebef2d1", "stdout_lines": ["f76e9e239faceca753ffd443c97bd3654ebef2d1"]} 2020-11-09T18:03:15.206 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:15 +0000 (0:00:00.162) 0:00:06.932 ******* 2020-11-09T18:03:15.215 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-11-09T18:03:15.215 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2020-11-09T18:03:15.227 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2020-11-09T18:03:15.229 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:15.265 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:15 +0000 (0:00:00.058) 0:00:06.991 ******* 2020-11-09T18:03:15.293 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2020-11-09T18:03:15.295 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:15.327 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:15 +0000 (0:00:00.061) 0:00:07.053 ******* 2020-11-09T18:03:15.334 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-11-09T18:03:15.336 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2020-11-09T18:03:15.357 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:15.391 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:15 +0000 (0:00:00.064) 0:00:07.117 ******* 2020-11-09T18:03:15.418 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2020-11-09T18:03:15.420 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:15.452 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:15 +0000 (0:00:00.060) 0:00:07.177 ******* 2020-11-09T18:03:15.479 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-11-09T18:03:15.481 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} TASK [users : Filter the managed_admin_users list] ***************************** skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:15.514 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:15 +0000 (0:00:00.062) 0:00:07.240 ******* 2020-11-09T18:03:15.615 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-11-09T18:03:15.617 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2020-11-09T18:03:15.648 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2020-11-09T18:03:15.650 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-09T18:03:15.682 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:15 +0000 (0:00:00.168) 0:00:07.408 ******* 2020-11-09T18:03:15.753 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-11-09T18:03:15.754 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2020-11-09T18:03:15.876 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2020-11-09T18:03:15.940 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2020-11-09T18:03:16.005 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2020-11-09T18:03:16.101 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2020-11-09T18:03:16.146 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2020-11-09T18:03:16.274 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2020-11-09T18:03:16.276 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2020-11-09T18:03:16.402 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2020-11-09T18:03:16.413 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2020-11-09T18:03:16.530 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2020-11-09T18:03:16.541 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2020-11-09T18:03:16.668 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2020-11-09T18:03:16.670 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2020-11-09T18:03:16.807 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2020-11-09T18:03:16.819 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2020-11-09T18:03:16.935 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2020-11-09T18:03:16.968 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2020-11-09T18:03:17.063 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2020-11-09T18:03:17.106 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2020-11-09T18:03:17.191 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2020-11-09T18:03:17.235 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2020-11-09T18:03:17.329 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2020-11-09T18:03:17.372 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2020-11-09T18:03:17.447 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2020-11-09T18:03:17.511 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2020-11-09T18:03:17.574 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2020-11-09T18:03:17.659 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2020-11-09T18:03:17.702 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2020-11-09T18:03:17.807 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2020-11-09T18:03:17.829 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2020-11-09T18:03:17.945 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/zyan", "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "move_home": false, "name": "zyan", "shell": "/bin/bash", "state": "present", "uid": 1122} 2020-11-09T18:03:17.967 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2020-11-09T18:03:18.074 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} 2020-11-09T18:03:18.086 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2020-11-09T18:03:18.203 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2020-11-09T18:03:18.225 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2020-11-09T18:03:18.351 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} 2020-11-09T18:03:18.353 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2020-11-09T18:03:18.486 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2020-11-09T18:03:18.496 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} 2020-11-09T18:03:18.611 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/zyan", "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "move_home": false, "name": "zyan", "shell": "/bin/bash", "state": "present", "uid": 1122} 2020-11-09T18:03:18.644 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2020-11-09T18:03:18.739 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} 2020-11-09T18:03:18.792 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2020-11-09T18:03:18.866 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2020-11-09T18:03:18.929 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} 2020-11-09T18:03:18.995 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} 2020-11-09T18:03:19.070 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2020-11-09T18:03:19.133 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} 2020-11-09T18:03:19.207 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} 2020-11-09T18:03:19.260 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2020-11-09T18:03:19.355 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} 2020-11-09T18:03:19.408 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2020-11-09T18:03:19.493 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} 2020-11-09T18:03:19.537 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} 2020-11-09T18:03:19.642 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1134} 2020-11-09T18:03:19.654 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2020-11-09T18:03:19.781 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1135} 2020-11-09T18:03:19.792 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} 2020-11-09T18:03:19.909 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1136} 2020-11-09T18:03:19.920 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} 2020-11-09T18:03:20.037 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1137} 2020-11-09T18:03:20.049 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} 2020-11-09T18:03:20.176 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1134} 2020-11-09T18:03:20.189 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1138} 2020-11-09T18:03:20.315 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1135} 2020-11-09T18:03:20.328 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1139} 2020-11-09T18:03:20.445 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1136} 2020-11-09T18:03:20.478 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2020-11-09T18:03:20.583 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1137} 2020-11-09T18:03:20.615 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1141} 2020-11-09T18:03:20.701 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1138} 2020-11-09T18:03:20.754 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1142} 2020-11-09T18:03:20.842 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1139} 2020-11-09T18:03:20.897 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1143} 2020-11-09T18:03:20.960 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2020-11-09T18:03:21.048 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1144} 2020-11-09T18:03:21.101 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1141} 2020-11-09T18:03:21.197 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1145} 2020-11-09T18:03:21.251 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1142} 2020-11-09T18:03:21.325 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1146} 2020-11-09T18:03:21.378 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1143} 2020-11-09T18:03:21.468 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1147} 2020-11-09T18:03:21.512 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1144} 2020-11-09T18:03:21.597 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2020-11-09T18:03:21.639 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1145} 2020-11-09T18:03:21.745 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1149} 2020-11-09T18:03:21.768 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1146} 2020-11-09T18:03:21.883 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1150} 2020-11-09T18:03:21.896 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1147} 2020-11-09T18:03:22.032 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2020-11-09T18:03:22.033 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1151} 2020-11-09T18:03:22.158 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1149} 2020-11-09T18:03:22.181 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1152} 2020-11-09T18:03:22.286 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1150} 2020-11-09T18:03:22.331 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1153} 2020-11-09T18:03:22.416 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1151} 2020-11-09T18:03:22.480 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1154} 2020-11-09T18:03:22.543 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1152} 2020-11-09T18:03:22.617 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1155} 2020-11-09T18:03:22.671 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1153} 2020-11-09T18:03:22.756 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1156} 2020-11-09T18:03:22.809 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1154} 2020-11-09T18:03:22.905 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1157} 2020-11-09T18:03:22.937 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1155} 2020-11-09T18:03:23.023 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1158} 2020-11-09T18:03:23.056 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1156} 2020-11-09T18:03:23.171 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1159} 2020-11-09T18:03:23.183 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1157} 2020-11-09T18:03:23.298 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1160} 2020-11-09T18:03:23.309 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1158} 2020-11-09T18:03:23.444 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1159} 2020-11-09T18:03:23.445 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1161} 2020-11-09T18:03:23.571 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1160} 2020-11-09T18:03:23.572 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1162} 2020-11-09T18:03:23.709 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1161} 2020-11-09T18:03:23.720 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1163} 2020-11-09T18:03:23.826 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1162} 2020-11-09T18:03:23.858 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1164} 2020-11-09T18:03:23.963 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1163} 2020-11-09T18:03:24.000 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2020-11-09T18:03:24.084 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1164} 2020-11-09T18:03:24.139 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2020-11-09T18:03:24.214 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2020-11-09T18:03:24.279 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1166} 2020-11-09T18:03:24.355 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2020-11-09T18:03:24.432 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1167} 2020-11-09T18:03:24.505 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1166} 2020-11-09T18:03:24.580 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1168} 2020-11-09T18:03:24.634 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1167} 2020-11-09T18:03:24.730 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1169} 2020-11-09T18:03:24.763 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1168} 2020-11-09T18:03:24.880 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1170} 2020-11-09T18:03:24.891 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1169} 2020-11-09T18:03:25.009 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1171} 2020-11-09T18:03:25.020 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1170} 2020-11-09T18:03:25.147 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1172} 2020-11-09T18:03:25.147 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1171} 2020-11-09T18:03:25.286 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1172} 2020-11-09T18:03:25.298 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1173} 2020-11-09T18:03:25.414 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1173} 2020-11-09T18:03:25.448 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1174} 2020-11-09T18:03:25.553 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1174} 2020-11-09T18:03:25.576 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1175} 2020-11-09T18:03:25.683 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1175} 2020-11-09T18:03:25.726 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1176} 2020-11-09T18:03:25.822 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1176} 2020-11-09T18:03:25.875 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1177} 2020-11-09T18:03:25.951 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1177} 2020-11-09T18:03:26.021 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1178} 2020-11-09T18:03:26.075 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1178} 2020-11-09T18:03:26.171 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1179} 2020-11-09T18:03:26.364 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1179} ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1180} ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1180} 2020-11-09T18:03:26.459 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1181} 2020-11-09T18:03:26.461 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1181} 2020-11-09T18:03:26.599 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1182} 2020-11-09T18:03:26.600 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1182} 2020-11-09T18:03:26.727 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1183} 2020-11-09T18:03:26.739 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1183} 2020-11-09T18:03:26.865 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1184} 2020-11-09T18:03:26.888 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1184} 2020-11-09T18:03:26.994 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1185} 2020-11-09T18:03:27.026 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1185} 2020-11-09T18:03:27.122 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1186} 2020-11-09T18:03:27.144 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1186} 2020-11-09T18:03:27.249 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1187} 2020-11-09T18:03:27.292 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1187} 2020-11-09T18:03:27.381 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1188} 2020-11-09T18:03:27.434 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1188} 2020-11-09T18:03:27.507 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1189} 2020-11-09T18:03:27.581 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1189} 2020-11-09T18:03:27.646 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1190} 2020-11-09T18:03:27.721 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1190} 2020-11-09T18:03:27.774 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1191} 2020-11-09T18:03:27.870 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1191} 2020-11-09T18:03:27.902 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1192} 2020-11-09T18:03:28.008 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1192} 2020-11-09T18:03:28.040 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1193} 2020-11-09T18:03:28.155 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1193} 2020-11-09T18:03:28.167 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1194} 2020-11-09T18:03:28.291 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1194} 2020-11-09T18:03:28.303 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1195} 2020-11-09T18:03:28.430 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1196} 2020-11-09T18:03:28.441 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1195} 2020-11-09T18:03:28.557 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1197} 2020-11-09T18:03:28.579 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1196} 2020-11-09T18:03:28.695 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1198} 2020-11-09T18:03:28.707 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1197} 2020-11-09T18:03:28.822 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1199} 2020-11-09T18:03:28.855 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1198} 2020-11-09T18:03:28.959 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1200} 2020-11-09T18:03:29.002 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1199} 2020-11-09T18:03:29.085 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1201} 2020-11-09T18:03:29.149 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1200} 2020-11-09T18:03:29.214 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2020-11-09T18:03:29.289 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1201} 2020-11-09T18:03:29.351 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1202} 2020-11-09T18:03:29.426 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2020-11-09T18:03:29.480 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1203} 2020-11-09T18:03:29.565 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1202} 2020-11-09T18:03:29.609 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1204} 2020-11-09T18:03:29.696 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1203} 2020-11-09T18:03:29.740 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1205} 2020-11-09T18:03:29.812 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1204} 2020-11-09T18:03:29.875 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1206} 2020-11-09T18:03:29.950 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1205} 2020-11-09T18:03:30.007 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1207} 2020-11-09T18:03:30.093 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1206} 2020-11-09T18:03:30.135 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1208} 2020-11-09T18:03:30.241 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1207} 2020-11-09T18:03:30.263 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1209} 2020-11-09T18:03:30.391 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1208} 2020-11-09T18:03:30.404 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1210} 2020-11-09T18:03:30.529 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1211} 2020-11-09T18:03:30.532 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1209} 2020-11-09T18:03:30.657 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1212} 2020-11-09T18:03:30.680 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1210} 2020-11-09T18:03:30.784 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1213} 2020-11-09T18:03:30.817 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1211} 2020-11-09T18:03:30.914 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1214} 2020-11-09T18:03:30.956 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1212} 2020-11-09T18:03:31.050 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1215} 2020-11-09T18:03:31.103 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1213} 2020-11-09T18:03:31.177 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1216} 2020-11-09T18:03:31.242 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1214} 2020-11-09T18:03:31.306 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1217} 2020-11-09T18:03:31.392 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1215} 2020-11-09T18:03:31.459 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1218} 2020-11-09T18:03:31.536 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1216} 2020-11-09T18:03:31.590 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1219} 2020-11-09T18:03:31.677 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1217} 2020-11-09T18:03:31.721 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1220} 2020-11-09T18:03:31.836 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1218} 2020-11-09T18:03:31.858 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1221} 2020-11-09T18:03:31.986 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1219} 2020-11-09T18:03:31.987 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1222} 2020-11-09T18:03:32.114 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1223} 2020-11-09T18:03:32.136 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1220} 2020-11-09T18:03:32.241 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1224} 2020-11-09T18:03:32.284 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1221} 2020-11-09T18:03:32.379 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1225} 2020-11-09T18:03:32.422 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1222} 2020-11-09T18:03:32.505 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1226} 2020-11-09T18:03:32.558 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1223} 2020-11-09T18:03:32.632 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1227} 2020-11-09T18:03:32.706 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1224} 2020-11-09T18:03:32.769 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1228} 2020-11-09T18:03:32.843 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1225} 2020-11-09T18:03:32.896 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1229} 2020-11-09T18:03:32.969 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1226} 2020-11-09T18:03:33.033 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1230} 2020-11-09T18:03:33.106 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1227} 2020-11-09T18:03:33.159 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1231} 2020-11-09T18:03:33.244 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1228} 2020-11-09T18:03:33.287 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1232} 2020-11-09T18:03:33.371 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1229} 2020-11-09T18:03:33.424 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1233} 2020-11-09T18:03:33.532 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:33 +0000 (0:00:17.850) 0:00:25.258 ******* 2020-11-09T18:03:33.541 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1230} 2020-11-09T18:03:33.554 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2020-11-09T18:03:33.587 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:33 +0000 (0:00:00.054) 0:00:25.313 ******* 2020-11-09T18:03:33.627 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2020-11-09T18:03:33.628 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2020-11-09T18:03:33.630 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2020-11-09T18:03:33.632 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancic 2020-11-09T18:03:33.633 INFO:teuthology.task.ansible.out:h", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkK 2020-11-09T18:03:33.635 INFO:teuthology.task.ansible.out:ztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 2020-11-09T18:03:33.636 INFO:teuthology.task.ansible.out:HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg 2020-11-09T18:03:33.638 INFO:teuthology.task.ansible.out:e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6 2020-11-09T18:03:33.639 INFO:teuthology.task.ansible.out:d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df 2020-11-09T18:03:33.641 INFO:teuthology.task.ansible.out:79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+ 2020-11-09T18:03:33.642 INFO:teuthology.task.ansible.out:7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b 2020-11-09T18:03:33.644 INFO:teuthology.task.ansible.out:0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/ 2020-11-09T18:03:33.645 INFO:teuthology.task.ansible.out:nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelope 2020-11-09T18:03:33.647 INFO:teuthology.task.ansible.out:z", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0 2020-11-09T18:03:33.649 INFO:teuthology.task.ansible.out:dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}]}, "changed": false} 2020-11-09T18:03:33.650 INFO:teuthology.task.ansible.out: 2020-11-09T18:03:33.690 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:33 +0000 (0:00:00.102) 0:00:25.415 ******* 2020-11-09T18:03:33.697 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-11-09T18:03:33.698 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1231} 2020-11-09T18:03:33.815 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1232} 2020-11-09T18:03:33.954 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1233} 2020-11-09T18:03:34.101 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:34 +0000 (0:00:00.412) 0:00:25.827 ******* 2020-11-09T18:03:34.130 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2020-11-09T18:03:34.165 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:34 +0000 (0:00:00.063) 0:00:25.891 ******* 2020-11-09T18:03:34.216 INFO:teuthology.task.ansible.out: 2020-11-09T18:03:34.218 INFO:teuthology.task.ansible.out:TASK [users : Merge managed_users and managed_admin users] ********************* 2020-11-09T18:03:34.220 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2020-11-09T18:03:34.221 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2020-11-09T18:03:34.223 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancic 2020-11-09T18:03:34.225 INFO:teuthology.task.ansible.out:h", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkK 2020-11-09T18:03:34.227 INFO:teuthology.task.ansible.out:ztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 2020-11-09T18:03:34.228 INFO:teuthology.task.ansible.out:HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg 2020-11-09T18:03:34.230 INFO:teuthology.task.ansible.out:e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6 2020-11-09T18:03:34.232 INFO:teuthology.task.ansible.out:d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c 2020-11-09T18:03:34.233 INFO:teuthology.task.ansible.out:08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0 2020-11-09T18:03:34.235 INFO:teuthology.task.ansible.out:FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f 2020-11-09T18:03:34.236 INFO:teuthology.task.ansible.out:756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b97150039029373 2020-11-09T18:03:34.238 INFO:teuthology.task.ansible.out:28801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devB 2020-11-09T18:03:34.239 INFO:teuthology.task.ansible.out:ox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40e 2020-11-09T18:03:34.241 INFO:teuthology.task.ansible.out:f3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}]}, "changed": false} 2020-11-09T18:03:34.241 INFO:teuthology.task.ansible.out: 2020-11-09T18:03:34.281 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:34 +0000 (0:00:00.116) 0:00:26.007 ******* 2020-11-09T18:03:34.521 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2020-11-09T18:03:34.523 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com -> localhost] => {"after": "3af9be36a63516879e86faffe49764ecad425070", "attempts": 1, "before": "3af9be36a63516879e86faffe49764ecad425070", "changed": false, "remote_url_changed": false} 2020-11-09T18:03:34.558 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:34 +0000 (0:00:00.276) 0:00:26.283 ******* 2020-11-09T18:03:34.747 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-11-09T18:03:34.748 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2020-11-09T18:03:34.852 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2020-11-09T18:03:34.854 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com -> localhost] => {"after": "3af9be36a63516879e86faffe49764ecad425070", "attempts": 1, "before": "3af9be36a63516879e86faffe49764ecad425070", "changed": false, "remote_url_changed": false} 2020-11-09T18:03:34.888 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:34 +0000 (0:00:00.330) 0:00:26.614 ******* 2020-11-09T18:03:34.895 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-11-09T18:03:34.897 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2020-11-09T18:03:35.012 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2020-11-09T18:03:35.087 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2020-11-09T18:03:35.149 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2020-11-09T18:03:35.151 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2020-11-09T18:03:35.212 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2020-11-09T18:03:35.297 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2020-11-09T18:03:35.350 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2020-11-09T18:03:35.423 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2020-11-09T18:03:35.425 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2020-11-09T18:03:35.507 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2020-11-09T18:03:35.509 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2020-11-09T18:03:35.539 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2020-11-09T18:03:35.634 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2020-11-09T18:03:35.676 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2020-11-09T18:03:35.771 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2020-11-09T18:03:35.773 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2020-11-09T18:03:35.803 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2020-11-09T18:03:35.815 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:35.921 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2020-11-09T18:03:35.933 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2020-11-09T18:03:35.934 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2020-11-09T18:03:36.069 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2020-11-09T18:03:36.071 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2020-11-09T18:03:36.080 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:36.081 INFO:teuthology.task.ansible.out: 2020-11-09T18:03:36.083 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2020-11-09T18:03:36.094 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:36.219 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2020-11-09T18:03:36.221 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2020-11-09T18:03:36.231 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:36.348 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2020-11-09T18:03:36.349 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2020-11-09T18:03:36.359 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2020-11-09T18:03:36.361 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2020-11-09T18:03:36.476 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2020-11-09T18:03:36.487 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2020-11-09T18:03:36.489 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2020-11-09T18:03:36.498 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:36.509 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:36.604 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2020-11-09T18:03:36.647 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2020-11-09T18:03:36.731 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2020-11-09T18:03:36.774 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2020-11-09T18:03:36.776 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2020-11-09T18:03:36.858 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2020-11-09T18:03:36.911 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2020-11-09T18:03:36.985 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2020-11-09T18:03:37.038 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2020-11-09T18:03:37.113 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9/fkAshmsXQfd6cJ+vEb0MZDaUT+OLmr1vAq0+H5g9w3hApJ2UEOFSaiz6ywgBAFeBeu/KmUALOhJIKKk0imoTunYclqxVPBK6tG/2b+ME8VE1afoEpMem3Bzfkrq0pDt6DLFBDlt9rRIqtO/ZlqEayWkEwTXJ7pwj+oJS03Z3Y7q0Iqw8PToKqJWwy2OvW1I9Pi0Q8s4YV9I34rSO7Wv4w43AXpFHRlvJWIcI5g5WFI5tnjel+kUAWMZft9ynFoAP5iXdByq897r4jAF74hapaxLMsHBz4MyYmPZqS76Cx2e3kGqXeetPinfcZxC3ViboR7o/1 root@zhyan-desk1", "key_options": null, "keyfile": "/home/zyan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zyan", "validate_certs": true} 2020-11-09T18:03:37.175 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2020-11-09T18:03:37.239 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2020-11-09T18:03:37.241 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2020-11-09T18:03:37.312 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2020-11-09T18:03:37.366 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2020-11-09T18:03:37.368 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2020-11-09T18:03:37.463 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2020-11-09T18:03:37.506 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2020-11-09T18:03:37.600 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9/fkAshmsXQfd6cJ+vEb0MZDaUT+OLmr1vAq0+H5g9w3hApJ2UEOFSaiz6ywgBAFeBeu/KmUALOhJIKKk0imoTunYclqxVPBK6tG/2b+ME8VE1afoEpMem3Bzfkrq0pDt6DLFBDlt9rRIqtO/ZlqEayWkEwTXJ7pwj+oJS03Z3Y7q0Iqw8PToKqJWwy2OvW1I9Pi0Q8s4YV9I34rSO7Wv4w43AXpFHRlvJWIcI5g5WFI5tnjel+kUAWMZft9ynFoAP5iXdByq897r4jAF74hapaxLMsHBz4MyYmPZqS76Cx2e3kGqXeetPinfcZxC3ViboR7o/1 root@zhyan-desk1", "key_options": null, "keyfile": "/home/zyan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zyan", "validate_certs": true} 2020-11-09T18:03:37.634 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2020-11-09T18:03:37.750 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2020-11-09T18:03:37.751 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2020-11-09T18:03:37.772 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2020-11-09T18:03:37.774 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2020-11-09T18:03:37.876 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2020-11-09T18:03:37.878 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2020-11-09T18:03:37.912 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2020-11-09T18:03:38.027 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2020-11-09T18:03:38.038 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2020-11-09T18:03:38.165 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2020-11-09T18:03:38.177 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2020-11-09T18:03:38.292 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2020-11-09T18:03:38.317 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2020-11-09T18:03:38.319 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2020-11-09T18:03:38.434 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2020-11-09T18:03:38.447 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2020-11-09T18:03:38.563 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2020-11-09T18:03:38.565 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2020-11-09T18:03:38.608 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2020-11-09T18:03:38.695 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2020-11-09T18:03:38.728 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2020-11-09T18:03:38.813 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2020-11-09T18:03:38.879 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2020-11-09T18:03:38.945 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2020-11-09T18:03:39.030 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2020-11-09T18:03:39.084 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2020-11-09T18:03:39.179 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2020-11-09T18:03:39.180 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2020-11-09T18:03:39.212 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2020-11-09T18:03:39.319 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2020-11-09T18:03:39.330 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2020-11-09T18:03:39.457 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-11-09T18:03:39.458 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2020-11-09T18:03:39.586 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2020-11-09T18:03:39.597 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2020-11-09T18:03:39.714 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2020-11-09T18:03:39.726 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2020-11-09T18:03:39.831 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2020-11-09T18:03:39.833 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2020-11-09T18:03:39.874 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2020-11-09T18:03:39.971 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2020-11-09T18:03:39.973 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2020-11-09T18:03:40.025 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2020-11-09T18:03:40.108 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2020-11-09T18:03:40.173 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-11-09T18:03:40.237 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2020-11-09T18:03:40.301 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2020-11-09T18:03:40.363 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2020-11-09T18:03:40.437 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2020-11-09T18:03:40.481 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2020-11-09T18:03:40.484 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2020-11-09T18:03:40.597 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2020-11-09T18:03:40.600 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2020-11-09T18:03:40.622 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2020-11-09T18:03:40.624 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2020-11-09T18:03:40.626 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2020-11-09T18:03:40.729 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2020-11-09T18:03:40.731 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2020-11-09T18:03:40.752 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2020-11-09T18:03:40.868 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2020-11-09T18:03:40.880 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2020-11-09T18:03:41.008 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2020-11-09T18:03:41.021 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2020-11-09T18:03:41.151 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2020-11-09T18:03:41.173 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2020-11-09T18:03:41.280 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2020-11-09T18:03:41.333 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2020-11-09T18:03:41.338 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2020-11-09T18:03:41.408 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2020-11-09T18:03:41.410 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2020-11-09T18:03:41.472 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2020-11-09T18:03:41.474 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2020-11-09T18:03:41.476 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2020-11-09T18:03:41.538 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2020-11-09T18:03:41.613 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2020-11-09T18:03:41.656 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2020-11-09T18:03:41.752 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2020-11-09T18:03:41.785 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2020-11-09T18:03:41.787 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2020-11-09T18:03:41.913 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2020-11-09T18:03:41.925 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2020-11-09T18:03:42.063 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2020-11-09T18:03:42.065 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2020-11-09T18:03:42.194 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2020-11-09T18:03:42.196 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2020-11-09T18:03:42.209 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2020-11-09T18:03:42.326 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2020-11-09T18:03:42.328 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2020-11-09T18:03:42.329 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2020-11-09T18:03:42.453 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2020-11-09T18:03:42.475 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2020-11-09T18:03:42.580 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2020-11-09T18:03:42.612 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2020-11-09T18:03:42.704 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-11-09T18:03:42.758 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2020-11-09T18:03:42.760 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2020-11-09T18:03:42.833 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2020-11-09T18:03:42.899 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2020-11-09T18:03:42.963 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2020-11-09T18:03:43.047 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2020-11-09T18:03:43.089 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2020-11-09T18:03:43.194 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2020-11-09T18:03:43.196 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2020-11-09T18:03:43.216 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2020-11-09T18:03:43.311 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2020-11-09T18:03:43.344 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2020-11-09T18:03:43.460 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2020-11-09T18:03:43.471 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2020-11-09T18:03:43.577 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2020-11-09T18:03:43.590 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2020-11-09T18:03:43.706 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-11-09T18:03:43.718 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2020-11-09T18:03:43.850 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKF 2020-11-09T18:03:43.853 INFO:teuthology.task.ansible.out:zu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2020-11-09T18:03:43.979 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2020-11-09T18:03:43.999 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2020-11-09T18:03:44.104 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2020-11-09T18:03:44.136 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2020-11-09T18:03:44.244 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2020-11-09T18:03:44.278 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2020-11-09T18:03:44.362 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2020-11-09T18:03:44.415 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2020-11-09T18:03:44.478 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC709gIxs1RgiA95ddp2uymzgV84VHJVVMj71FyTlz6pa0gtlaxF+r16p8dnNWjhyJRUXMwBKFOwHx95bcdfiRA+BfC04UnI8VvBxoED5Nt9EbKD5h78ZoNEeQJKoG1dsM/kWWlTd9WCsCyzjRz32+YNWul0LOhrfloHP+LHnfjf/uOnzd8iRHgnpWr5Z9uB0MBu4y/L6AmHnR+8pMZQq6Bfno8ZLwNCzx9v+4MqOz3jY7FYv7SJIftQH0fJygIfCz3bKyUEWk9a/JJacETuJGBnHgNAXUr/FtM1IFiMl2SsTixRgnosEz4pX8EIe5LJyw/Qudro8BPEC8aIGy0XAkerGmTt5iALMnzmeeanZCFm4rF/YiFoHLMS/B0uv42MD64hB3knrV8Bcq+mRjwVOYgecO+xoiBsU6FZeXaBk5IsjmYTwataGv3ILEmNpss0WeqHxF9nd7oTLnw51cZfcaerOnJ5IZJZTXwLNVr7K176BunHggAhjvtvwdeKhphqt9f3ilLr6gzksEoAR4JBOYJJtd8sEqHwBtti+YbMGwH9Y+Z+SujlQBQIsom60xj5OzrwjsIi5JAtOhbc1Mq2fptSXRsR8ZBT8aUFqZ8R7ixEqq2F47OAznbhSQKidzwUNkWNI3WAKi38Oi4aya1k3iqtL04bGf8R/xD5Vq/2J+UlQ== vant alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2020-11-09T18:03:44.563 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2020-11-09T18:03:44.613 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2020-11-09T18:03:44.698 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2020-11-09T18:03:44.754 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2020-11-09T18:03:44.839 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2020-11-09T18:03:44.884 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2020-11-09T18:03:44.886 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2020-11-09T18:03:44.981 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2020-11-09T18:03:45.022 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2020-11-09T18:03:45.116 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2020-11-09T18:03:45.150 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2020-11-09T18:03:45.256 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2020-11-09T18:03:45.268 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2020-11-09T18:03:45.406 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht 2020-11-09T18:03:45.407 INFO:teuthology.task.ansible.out:5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2020-11-09T18:03:45.522 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2020-11-09T18:03:45.524 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2020-11-09T18:03:45.544 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2020-11-09T18:03:45.656 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2020-11-09T18:03:45.658 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2020-11-09T18:03:45.678 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC709gIxs1RgiA95ddp2uymzgV84VHJVVMj71FyTlz6pa0gtlaxF+r16p8dnNWjhyJRUXMwBKFOwHx95bcdfiRA+BfC04UnI8VvBxoED5Nt9EbKD5h78ZoNEeQJKoG1dsM/kWWlTd9WCsCyzjRz32+YNWul0LOhrfloHP+LHnfjf/uOnzd8iRHgnpWr5Z9uB0MBu4y/L6AmHnR+8pMZQq6Bfno8ZLwNCzx9v+4MqOz3jY7FYv7SJIftQH0fJygIfCz3bKyUEWk9a/JJacETuJGBnHgNAXUr/FtM1IFiMl2SsTixRgnosEz4pX8EIe5LJyw/Qudro8BPEC8aIGy0XAkerGmTt5iALMnzmeeanZCFm4rF/YiFoHLMS/B0uv42MD64hB3knrV8Bcq+mRjwVOYgecO+xoiBsU6FZeXaBk5IsjmYTwataGv3ILEmNpss0WeqHxF9nd7oTLnw51cZfcaerOnJ5IZJZTXwLNVr7K176BunHggAhjvtvwdeKhphqt9f3ilLr6gzksEoAR4JBOYJJtd8sEqHwBtti+YbMGwH9Y+Z+SujlQBQIsom60xj5OzrwjsIi5JAtOhbc1Mq2fptSXRsR8ZBT8aUFqZ8R7ixEqq2F47OAznbhSQKidzwUNkWNI3WAKi38Oi4aya1k3iqtL04bGf8R/xD5Vq/2J+UlQ== vant alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2020-11-09T18:03:45.784 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2020-11-09T18:03:45.828 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2020-11-09T18:03:45.903 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2020-11-09T18:03:45.970 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2020-11-09T18:03:46.033 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1q3k4RvLLCaG874GDJjja4eYqFwrMjsTU+aRzambaVuCmAJT3KQMGuwMjTuCIN7R1Fqr+/dxUClamsfsGqPMe0ASHT9X/0AyNgEkr6na1Fvk9NAwAFelwtiRIHhclzYf6KY8QHZN/eBy43prVKDXd30b15gwQyVqU0/xmny0GrYCN24k3N/R3divt2GVqptYgEwS2KEyFT5EMnJWjdIVG9mBOHrgec2VF/rc6w3BZ0s56iKdpOeCgk5ISYTcom+Xrf+fSDHirlviZv05tBP8ESmampUT/EnwF8Mdaln5XJvZPhrq2JpUisi9fVeR+zlzg6gRGubvokdA3AERrhCQ9 skoduri@rgw_vm", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2020-11-09T18:03:46.075 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2020-11-09T18:03:46.077 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2020-11-09T18:03:46.164 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2020-11-09T18:03:46.217 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2020-11-09T18:03:46.291 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2020-11-09T18:03:46.365 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2020-11-09T18:03:46.418 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2020-11-09T18:03:46.503 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2020-11-09T18:03:46.535 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2020-11-09T18:03:46.631 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2020-11-09T18:03:46.677 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2020-11-09T18:03:46.680 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2020-11-09T18:03:46.773 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2020-11-09T18:03:46.775 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2020-11-09T18:03:46.807 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2020-11-09T18:03:46.913 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2020-11-09T18:03:46.915 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2020-11-09T18:03:46.925 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2020-11-09T18:03:47.052 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2020-11-09T18:03:47.069 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2020-11-09T18:03:47.197 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2020-11-09T18:03:47.201 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2020-11-09T18:03:47.329 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2020-11-09T18:03:47.351 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1q3k4RvLLCaG874GDJjja4eYqFwrMjsTU+aRzambaVuCmAJT3KQMGuwMjTuCIN7R1Fqr+/dxUClamsfsGqPMe0ASHT9X/0AyNgEkr6na1Fvk9NAwAFelwtiRIHhclzYf6KY8QHZN/eBy43prVKDXd30b15gwQyVqU0/xmny0GrYCN24k3N/R3divt2GVqptYgEwS2KEyFT5EMnJWjdIVG9mBOHrgec2VF/rc6w3BZ0s56iKdpOeCgk5ISYTcom+Xrf+fSDHirlviZv05tBP8ESmampUT/EnwF8Mdaln5XJvZPhrq2JpUisi9fVeR+zlzg6gRGubvokdA3AERrhCQ9 skoduri@rgw_vm", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2020-11-09T18:03:47.459 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2020-11-09T18:03:47.502 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2020-11-09T18:03:47.594 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", 2020-11-09T18:03:47.595 INFO:teuthology.task.ansible.out:"manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2020-11-09T18:03:47.626 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2020-11-09T18:03:47.723 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2020-11-09T18:03:47.776 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2020-11-09T18:03:47.852 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2020-11-09T18:03:47.916 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2020-11-09T18:03:47.969 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2020-11-09T18:03:48.033 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2020-11-09T18:03:48.034 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2020-11-09T18:03:48.106 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2020-11-09T18:03:48.161 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2020-11-09T18:03:48.246 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2020-11-09T18:03:48.290 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2020-11-09T18:03:48.386 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2020-11-09T18:03:48.419 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2020-11-09T18:03:48.512 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2020-11-09T18:03:48.546 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2020-11-09T18:03:48.640 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2020-11-09T18:03:48.688 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2020-11-09T18:03:48.762 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2020-11-09T18:03:48.837 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2020-11-09T18:03:48.890 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2020-11-09T18:03:48.964 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", 2020-11-09T18:03:48.965 INFO:teuthology.task.ansible.out:"manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2020-11-09T18:03:49.017 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2020-11-09T18:03:49.092 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2020-11-09T18:03:49.146 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2020-11-09T18:03:49.241 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2020-11-09T18:03:49.274 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2020-11-09T18:03:49.371 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2020-11-09T18:03:49.402 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2020-11-09T18:03:49.509 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2020-11-09T18:03:49.531 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2020-11-09T18:03:49.651 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2020-11-09T18:03:49.663 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2020-11-09T18:03:49.779 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2020-11-09T18:03:49.791 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2020-11-09T18:03:49.907 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2020-11-09T18:03:49.909 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2020-11-09T18:03:50.036 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2020-11-09T18:03:50.038 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2020-11-09T18:03:50.048 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2020-11-09T18:03:50.163 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2020-11-09T18:03:50.185 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2020-11-09T18:03:50.293 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2020-11-09T18:03:50.335 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2020-11-09T18:03:50.431 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2020-11-09T18:03:50.463 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2020-11-09T18:03:50.548 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2020-11-09T18:03:50.581 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2020-11-09T18:03:50.678 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2020-11-09T18:03:50.727 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2020-11-09T18:03:50.812 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2020-11-09T18:03:50.866 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2020-11-09T18:03:50.867 INFO:teuthology.task.ansible.out: 2020-11-09T18:03:50.931 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2020-11-09T18:03:50.994 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2020-11-09T18:03:51.057 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2020-11-09T18:03:51.131 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2020-11-09T18:03:51.184 INFO:teuthology.task.ansible.out:changed: [smithi051.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2020-11-09T18:03:51.258 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2020-11-09T18:03:51.321 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2020-11-09T18:03:51.406 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2020-11-09T18:03:51.449 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2020-11-09T18:03:51.536 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2020-11-09T18:03:51.537 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2020-11-09T18:03:51.576 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2020-11-09T18:03:51.671 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2020-11-09T18:03:51.716 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2020-11-09T18:03:51.795 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:51 +0000 (0:00:16.906) 0:00:43.520 ******* 2020-11-09T18:03:51.824 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2020-11-09T18:03:51.837 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-11-09T18:03:51.838 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:51.849 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:51.860 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:51.871 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:51.873 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:51.884 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:51.895 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:51.950 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-11-09T18:03:51.952 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2020-11-09T18:03:52.037 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2020-11-09T18:03:52.048 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.060 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.072 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-11-09T18:03:52.075 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2020-11-09T18:03:52.220 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLF 2020-11-09T18:03:52.222 INFO:teuthology.task.ansible.out:e1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2020-11-09T18:03:52.386 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-11-09T18:03:52.391 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2020-11-09T18:03:52.403 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** ok: [smithi051.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2020-11-09T18:03:52.414 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.416 INFO:teuthology.task.ansible.out: skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.428 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.431 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.441 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.452 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.455 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.464 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.468 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.477 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.489 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.500 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.511 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.514 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.524 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-11-09T18:03:52.526 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.539 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.549 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.551 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.563 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.565 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.576 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.578 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.587 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.599 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.600 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.610 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.621 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.624 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.633 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.644 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.648 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.659 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2020-11-09T18:03:52.668 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.670 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.681 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.692 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.695 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.711 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.724 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.729 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.735 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.738 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.747 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.760 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.772 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.780 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.783 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.794 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.805 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.808 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi078.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2020-11-09T18:03:52.819 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.820 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.831 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.843 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.845 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.854 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.865 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.876 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.878 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.888 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.890 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.899 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.901 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.911 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.913 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.922 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.933 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.935 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.945 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.946 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.957 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys using the keys repo] ********************** 2020-11-09T18:03:52.958 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2020-11-09T18:03:52.968 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-11-09T18:03:52.970 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.980 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.982 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:52.991 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.002 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.004 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.014 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.025 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.026 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.038 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.040 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.049 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.051 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.072 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.073 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.083 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.094 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2020-11-09T18:03:53.096 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.106 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.108 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.118 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.129 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.131 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.140 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.142 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.151 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.162 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.164 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.174 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.185 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.186 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.196 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.198 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.208 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.218 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.220 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.220 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2020-11-09T18:03:53.231 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-11-09T18:03:53.234 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.243 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.254 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.255 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.265 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.276 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.324 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:53 +0000 (0:00:01.529) 0:00:45.049 ******* 2020-11-09T18:03:53.351 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2020-11-09T18:03:53.352 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.387 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:53 +0000 (0:00:00.063) 0:00:45.112 ******* 2020-11-09T18:03:53.394 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-11-09T18:03:53.395 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2020-11-09T18:03:53.468 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:53 +0000 (0:00:00.081) 0:00:45.194 ******* 2020-11-09T18:03:53.507 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-11-09T18:03:53.509 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.519 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.530 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.530 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.542 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.543 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi051.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.554 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.565 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.567 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.669 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-11-09T18:03:53.671 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2020-11-09T18:03:53.723 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2020-11-09T18:03:53.733 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.735 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:53.797 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-11-09T18:03:53.799 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2020-11-09T18:03:53.872 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-11-09T18:03:53.873 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2020-11-09T18:03:53.914 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-11-09T18:03:53.916 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2020-11-09T18:03:54.009 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-11-09T18:03:54.011 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2020-11-09T18:03:54.021 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.032 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.043 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-11-09T18:03:54.045 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.054 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.056 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.066 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.077 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.077 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.088 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.099 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.101 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.110 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.121 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.123 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.132 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.143 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.145 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.154 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.166 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.166 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-11-09T18:03:54.166 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2020-11-09T18:03:54.168 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.178 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.189 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.200 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.211 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.212 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.222 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.234 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.245 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.256 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.267 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.269 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.278 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.289 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.291 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** ok: [smithi051.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2020-11-09T18:03:54.301 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-11-09T18:03:54.303 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.312 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.314 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.323 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.325 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.336 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.347 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.348 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.358 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.369 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.370 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.382 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.393 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.394 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.404 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.415 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.417 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi051.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.427 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.429 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.438 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.449 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.451 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.460 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.471 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.473 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.482 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.484 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.494 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.505 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.506 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.516 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.527 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.529 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.530 INFO:teuthology.task.ansible.out: 2020-11-09T18:03:54.541 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.543 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi051.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.553 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.564 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.566 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.576 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.577 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.597 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.599 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.609 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.620 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.621 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.631 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.632 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.643 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.654 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.655 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.665 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-11-09T18:03:54.667 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.677 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.678 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.689 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.690 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.700 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.711 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.713 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.727 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.738 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.739 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.749 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.761 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.762 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.775 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.787 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.788 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** ok: [smithi051.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2020-11-09T18:03:54.799 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-11-09T18:03:54.801 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.811 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.812 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.833 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.834 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.834 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.845 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.856 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.858 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.868 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.869 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.871 INFO:teuthology.task.ansible.out: 2020-11-09T18:03:54.881 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.882 INFO:teuthology.task.ansible.out: 2020-11-09T18:03:54.892 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.894 INFO:teuthology.task.ansible.out: skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:54.943 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:54 +0000 (0:00:01.474) 0:00:46.669 ******* 2020-11-09T18:03:54.951 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-11-09T18:03:54.953 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2020-11-09T18:03:54.974 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:55.007 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:55 +0000 (0:00:00.064) 0:00:46.733 ******* 2020-11-09T18:03:55.042 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-11-09T18:03:55.044 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2020-11-09T18:03:55.158 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2020-11-09T18:03:55.190 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2020-11-09T18:03:55.192 INFO:teuthology.task.ansible.out: 2020-11-09T18:03:55.287 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2020-11-09T18:03:55.330 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2020-11-09T18:03:55.404 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2020-11-09T18:03:55.456 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2020-11-09T18:03:55.531 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2020-11-09T18:03:55.605 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2020-11-09T18:03:55.659 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2020-11-09T18:03:55.722 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2020-11-09T18:03:55.776 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2020-11-09T18:03:55.839 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2020-11-09T18:03:55.841 INFO:teuthology.task.ansible.out: 2020-11-09T18:03:55.903 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2020-11-09T18:03:55.978 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2020-11-09T18:03:56.020 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2020-11-09T18:03:56.105 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2020-11-09T18:03:56.148 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2020-11-09T18:03:56.231 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2020-11-09T18:03:56.273 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2020-11-09T18:03:56.348 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2020-11-09T18:03:56.401 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2020-11-09T18:03:56.475 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2020-11-09T18:03:56.529 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2020-11-09T18:03:56.623 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2020-11-09T18:03:56.666 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2020-11-09T18:03:56.741 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2020-11-09T18:03:56.784 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2020-11-09T18:03:56.869 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2020-11-09T18:03:56.902 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2020-11-09T18:03:56.904 INFO:teuthology.task.ansible.out: 2020-11-09T18:03:57.009 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2020-11-09T18:03:57.020 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2020-11-09T18:03:57.139 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2020-11-09T18:03:57.150 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2020-11-09T18:03:57.266 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2020-11-09T18:03:57.277 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2020-11-09T18:03:57.403 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2020-11-09T18:03:57.460 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2020-11-09T18:03:57.533 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2020-11-09T18:03:57.555 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2020-11-09T18:03:57.649 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2020-11-09T18:03:57.681 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2020-11-09T18:03:57.778 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2020-11-09T18:03:57.821 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2020-11-09T18:03:57.895 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2020-11-09T18:03:57.958 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2020-11-09T18:03:58.021 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2020-11-09T18:03:58.075 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2020-11-09T18:03:58.139 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2020-11-09T18:03:58.205 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2020-11-09T18:03:58.268 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2020-11-09T18:03:58.341 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2020-11-09T18:03:58.394 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2020-11-09T18:03:58.470 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2020-11-09T18:03:58.522 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2020-11-09T18:03:58.586 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2020-11-09T18:03:58.652 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2020-11-09T18:03:58.694 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2020-11-09T18:03:58.769 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2020-11-09T18:03:58.802 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2020-11-09T18:03:58.887 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2020-11-09T18:03:58.939 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2020-11-09T18:03:59.015 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2020-11-09T18:03:59.067 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2020-11-09T18:03:59.131 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2020-11-09T18:03:59.179 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2020-11-09T18:03:59.263 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2020-11-09T18:03:59.316 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:59 +0000 (0:00:04.309) 0:00:51.042 ******* 2020-11-09T18:03:59.324 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2020-11-09T18:03:59.443 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2020-11-09T18:03:59.591 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2020-11-09T18:03:59.665 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2020-11-09T18:03:59.667 INFO:teuthology.task.ansible.out:changed: [smithi051.front.sepia.ceph.com] => {"changed": true, "checksum": "c87b5a8671ca71c661f8c33e2a2def18c9932825", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "392e33fe3d702764a3a1ed34f241b107", "mode": "0644", "owner": "root", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1604945039.3381586-99346999793285/source", "state": "file", "uid": 0} 2020-11-09T18:03:59.700 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:59 +0000 (0:00:00.383) 0:00:51.426 ******* 2020-11-09T18:03:59.728 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-11-09T18:03:59.730 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} TASK [common : Log the OS name, version and release] *************************** ok: [smithi051.front.sepia.ceph.com] => { "msg": "Host smithi051.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2020-11-09T18:03:59.762 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:59 +0000 (0:00:00.062) 0:00:51.488 ******* 2020-11-09T18:03:59.800 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2020-11-09T18:03:59.802 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2020-11-09T18:03:59.834 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:59 +0000 (0:00:00.071) 0:00:51.560 ******* 2020-11-09T18:03:59.841 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-11-09T18:03:59.843 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2020-11-09T18:03:59.864 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2020-11-09T18:03:59.866 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:59.897 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:59 +0000 (0:00:00.063) 0:00:51.623 ******* 2020-11-09T18:03:59.924 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2020-11-09T18:03:59.926 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:03:59.958 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:03:59 +0000 (0:00:00.060) 0:00:51.683 ******* 2020-11-09T18:03:59.985 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2020-11-09T18:03:59.987 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** ok: [smithi078.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2020-11-09T18:04:00.024 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:00 +0000 (0:00:00.066) 0:00:51.750 ******* 2020-11-09T18:04:00.052 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2020-11-09T18:04:00.053 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:00.088 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:00 +0000 (0:00:00.063) 0:00:51.813 ******* 2020-11-09T18:04:00.114 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2020-11-09T18:04:00.116 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** ok: [smithi078.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2020-11-09T18:04:00.148 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:00 +0000 (0:00:00.060) 0:00:51.874 ******* 2020-11-09T18:04:00.175 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2020-11-09T18:04:00.177 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:00.208 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:00 +0000 (0:00:00.059) 0:00:51.934 ******* 2020-11-09T18:04:00.235 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2020-11-09T18:04:00.237 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:00.270 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:00 +0000 (0:00:00.061) 0:00:51.996 ******* 2020-11-09T18:04:00.278 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-11-09T18:04:00.280 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2020-11-09T18:04:00.291 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2020-11-09T18:04:00.293 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:00.330 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:00 +0000 (0:00:00.059) 0:00:52.056 ******* 2020-11-09T18:04:00.358 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2020-11-09T18:04:00.359 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi051.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-09T18:04:00.392 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:00 +0000 (0:00:00.062) 0:00:52.118 ******* 2020-11-09T18:04:00.399 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-11-09T18:04:00.401 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2020-11-09T18:04:00.421 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:00.456 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:00 +0000 (0:00:00.063) 0:00:52.181 ******* 2020-11-09T18:04:00.483 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2020-11-09T18:04:00.485 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:00.517 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:00 +0000 (0:00:00.061) 0:00:52.243 ******* 2020-11-09T18:04:00.525 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-11-09T18:04:00.527 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2020-11-09T18:04:00.537 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** skipping: [smithi051.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-09T18:04:00.571 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:00 +0000 (0:00:00.053) 0:00:52.297 ******* 2020-11-09T18:04:00.598 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:00.635 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:00 +0000 (0:00:00.064) 0:00:52.361 ******* 2020-11-09T18:04:00.642 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-11-09T18:04:00.644 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2020-11-09T18:04:00.665 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2020-11-09T18:04:00.666 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-09T18:04:00.700 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:00 +0000 (0:00:00.064) 0:00:52.426 ******* 2020-11-09T18:04:00.727 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:00.763 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:00 +0000 (0:00:00.063) 0:00:52.489 ******* 2020-11-09T18:04:00.774 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-11-09T18:04:00.775 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2020-11-09T18:04:00.796 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2020-11-09T18:04:00.798 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:00.834 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:00 +0000 (0:00:00.070) 0:00:52.559 ******* 2020-11-09T18:04:00.861 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2020-11-09T18:04:00.863 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:00.896 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:00 +0000 (0:00:00.062) 0:00:52.622 ******* 2020-11-09T18:04:00.904 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi078.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2020-11-09T18:04:00.927 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:00.962 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:00 +0000 (0:00:00.065) 0:00:52.687 ******* 2020-11-09T18:04:00.989 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2020-11-09T18:04:00.990 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-09T18:04:01.028 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:01 +0000 (0:00:00.066) 0:00:52.753 ******* 2020-11-09T18:04:01.045 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-11-09T18:04:01.047 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2020-11-09T18:04:01.057 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-11-09T18:04:01.058 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:01.091 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:01 +0000 (0:00:00.063) 0:00:52.817 ******* 2020-11-09T18:04:01.120 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:01.154 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:01 +0000 (0:00:00.062) 0:00:52.879 ******* 2020-11-09T18:04:01.171 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-11-09T18:04:01.173 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2020-11-09T18:04:01.192 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2020-11-09T18:04:01.193 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:01.228 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:01 +0000 (0:00:00.074) 0:00:52.954 ******* 2020-11-09T18:04:01.261 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:01 +0000 (0:00:00.033) 0:00:52.987 ******* 2020-11-09T18:04:01.268 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2020-11-09T18:04:01.270 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:01.306 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:01 +0000 (0:00:00.044) 0:00:53.032 ******* 2020-11-09T18:04:01.335 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:01.369 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:01 +0000 (0:00:00.062) 0:00:53.094 ******* 2020-11-09T18:04:01.396 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2020-11-09T18:04:01.397 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:01.430 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:01 +0000 (0:00:00.061) 0:00:53.156 ******* 2020-11-09T18:04:01.458 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2020-11-09T18:04:01.460 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-09T18:04:01.492 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:01 +0000 (0:00:00.061) 0:00:53.218 ******* 2020-11-09T18:04:01.519 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-11-09T18:04:01.521 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:01.554 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:01 +0000 (0:00:00.061) 0:00:53.280 ******* 2020-11-09T18:04:01.586 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2020-11-09T18:04:01.588 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:01.620 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:01 +0000 (0:00:00.066) 0:00:53.346 ******* 2020-11-09T18:04:01.627 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2020-11-09T18:04:01.629 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => {"changed": true, "checksum": "c87b5a8671ca71c661f8c33e2a2def18c9932825", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "392e33fe3d702764a3a1ed34f241b107", "mode": "0644", "owner": "root", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1604945041.2833562-41553736918377/source", "state": "file", "uid": 0} 2020-11-09T18:04:01.662 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:01 +0000 (0:00:00.041) 0:00:53.388 ******* 2020-11-09T18:04:01.670 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-11-09T18:04:01.671 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:01.704 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:01 +0000 (0:00:00.042) 0:00:53.430 ******* 2020-11-09T18:04:01.711 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2020-11-09T18:04:01.713 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => { "msg": "Host smithi078.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2020-11-09T18:04:01.747 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:01 +0000 (0:00:00.043) 0:00:53.473 ******* 2020-11-09T18:04:01.754 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-11-09T18:04:01.756 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} skipping: [smithi051.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:01.792 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:01 +0000 (0:00:00.044) 0:00:53.518 ******* 2020-11-09T18:04:01.799 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2020-11-09T18:04:01.801 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2020-11-09T18:04:01.837 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:01 +0000 (0:00:00.045) 0:00:53.563 ******* 2020-11-09T18:04:01.844 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2020-11-09T18:04:01.845 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi051.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2020-11-09T18:04:01.879 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:01 +0000 (0:00:00.041) 0:00:53.605 ******* 2020-11-09T18:04:01.886 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2020-11-09T18:04:01.887 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:01.920 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:01 +0000 (0:00:00.040) 0:00:53.646 ******* 2020-11-09T18:04:01.926 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-11-09T18:04:01.928 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:01.961 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:01 +0000 (0:00:00.041) 0:00:53.687 ******* 2020-11-09T18:04:01.969 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:02.004 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:02 +0000 (0:00:00.042) 0:00:53.729 ******* 2020-11-09T18:04:02.033 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2020-11-09T18:04:02.035 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:02.069 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:02 +0000 (0:00:00.065) 0:00:53.795 ******* 2020-11-09T18:04:02.096 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2020-11-09T18:04:02.097 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:02.129 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:02 +0000 (0:00:00.060) 0:00:53.855 ******* 2020-11-09T18:04:02.136 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-11-09T18:04:02.137 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.015250", "end": "2020-11-09 18:04:01.754053", "rc": 0, "start": "2020-11-09 18:04:01.738803", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-11-09T18:04:02.171 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:02 +0000 (0:00:00.041) 0:00:53.897 ******* 2020-11-09T18:04:02.178 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2020-11-09T18:04:02.179 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:02.216 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:02 +0000 (0:00:00.045) 0:00:53.942 ******* 2020-11-09T18:04:02.223 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2020-11-09T18:04:02.225 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:02.258 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:02 +0000 (0:00:00.042) 0:00:53.984 ******* 2020-11-09T18:04:02.265 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2020-11-09T18:04:02.266 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:02.301 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:02 +0000 (0:00:00.043) 0:00:54.027 ******* 2020-11-09T18:04:02.308 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2020-11-09T18:04:02.310 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:02.343 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:02 +0000 (0:00:00.041) 0:00:54.069 ******* 2020-11-09T18:04:02.351 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2020-11-09T18:04:02.352 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:02.389 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:02 +0000 (0:00:00.045) 0:00:54.115 ******* 2020-11-09T18:04:02.396 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-11-09T18:04:02.398 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:02.436 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:02 +0000 (0:00:00.046) 0:00:54.161 ******* 2020-11-09T18:04:02.442 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2020-11-09T18:04:02.444 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:02.480 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:02 +0000 (0:00:00.044) 0:00:54.206 ******* 2020-11-09T18:04:02.487 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:02.527 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:02 +0000 (0:00:00.046) 0:00:54.252 ******* 2020-11-09T18:04:02.536 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2020-11-09T18:04:02.538 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi078.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-09T18:04:02.576 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:02 +0000 (0:00:00.049) 0:00:54.302 ******* 2020-11-09T18:04:02.584 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2020-11-09T18:04:02.586 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:02.618 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:02 +0000 (0:00:00.042) 0:00:54.344 ******* 2020-11-09T18:04:02.625 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2020-11-09T18:04:02.627 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:02.662 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:02 +0000 (0:00:00.043) 0:00:54.387 ******* 2020-11-09T18:04:02.668 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2020-11-09T18:04:02.669 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:02.703 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:02 +0000 (0:00:00.041) 0:00:54.429 ******* 2020-11-09T18:04:02.710 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2020-11-09T18:04:02.711 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:02.745 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:02 +0000 (0:00:00.041) 0:00:54.471 ******* 2020-11-09T18:04:02.751 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-11-09T18:04:02.753 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:02.787 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:02 +0000 (0:00:00.041) 0:00:54.512 ******* 2020-11-09T18:04:02.796 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2020-11-09T18:04:02.798 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-09T18:04:02.833 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:02 +0000 (0:00:00.046) 0:00:54.559 ******* 2020-11-09T18:04:02.869 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2020-11-09T18:04:02.870 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:02.908 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:02 +0000 (0:00:00.075) 0:00:54.634 ******* 2020-11-09T18:04:02.939 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** skipping: [smithi078.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-09T18:04:02.972 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:02 +0000 (0:00:00.064) 0:00:54.698 ******* 2020-11-09T18:04:03.000 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2020-11-09T18:04:03.001 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:03.033 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:03 +0000 (0:00:00.060) 0:00:54.759 ******* 2020-11-09T18:04:03.061 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2020-11-09T18:04:03.062 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:03.094 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:03 +0000 (0:00:00.060) 0:00:54.820 ******* 2020-11-09T18:04:03.121 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:03.155 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:03 +0000 (0:00:00.061) 0:00:54.881 ******* 2020-11-09T18:04:03.183 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2020-11-09T18:04:03.185 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:03.216 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:03 +0000 (0:00:00.060) 0:00:54.942 ******* 2020-11-09T18:04:03.244 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* skipping: [smithi078.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-09T18:04:03.278 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:03 +0000 (0:00:00.061) 0:00:55.003 ******* 2020-11-09T18:04:03.305 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-11-09T18:04:03.307 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:03.338 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:03 +0000 (0:00:00.060) 0:00:55.064 ******* 2020-11-09T18:04:03.367 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:03.400 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:03 +0000 (0:00:00.062) 0:00:55.126 ******* 2020-11-09T18:04:03.431 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2020-11-09T18:04:03.433 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:03.465 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:03 +0000 (0:00:00.064) 0:00:55.191 ******* 2020-11-09T18:04:03.493 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2020-11-09T18:04:03.495 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:03.526 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:03 +0000 (0:00:00.061) 0:00:55.252 ******* 2020-11-09T18:04:03.554 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2020-11-09T18:04:03.556 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:03.587 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:03 +0000 (0:00:00.060) 0:00:55.313 ******* 2020-11-09T18:04:03.619 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2020-11-09T18:04:03.620 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:03.662 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:03 +0000 (0:00:00.074) 0:00:55.387 ******* 2020-11-09T18:04:03.689 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2020-11-09T18:04:03.690 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-09T18:04:03.722 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:03 +0000 (0:00:00.060) 0:00:55.448 ******* 2020-11-09T18:04:03.749 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:03.783 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:03 +0000 (0:00:00.060) 0:00:55.508 ******* 2020-11-09T18:04:03.810 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:03.844 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:03 +0000 (0:00:00.061) 0:00:55.570 ******* 2020-11-09T18:04:03.861 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2020-11-09T18:04:03.863 INFO:teuthology.task.ansible.out:changed: [smithi051.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.935650", "end": "2020-11-09 18:04:03.500407", "rc": 0, "start": "2020-11-09 18:04:02.564757", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-09T18:04:03.896 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:03 +0000 (0:00:00.052) 0:00:55.622 ******* 2020-11-09T18:04:03.903 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-11-09T18:04:03.905 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:03.938 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:03 +0000 (0:00:00.041) 0:00:55.664 ******* 2020-11-09T18:04:03.945 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-11-09T18:04:03.947 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:03.982 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:03 +0000 (0:00:00.043) 0:00:55.708 ******* 2020-11-09T18:04:03.989 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-11-09T18:04:03.991 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} skipping: [smithi078.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:04.028 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:04 +0000 (0:00:00.045) 0:00:55.753 ******* 2020-11-09T18:04:04.067 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2020-11-09T18:04:04.069 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi078.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2020-11-09T18:04:04.102 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:04 +0000 (0:00:00.074) 0:00:55.828 ******* 2020-11-09T18:04:04.129 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-11-09T18:04:04.131 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [common : Clean apt cache] ************************************************ [WARNING]: Consider using the apt module rather than running 'apt-get'. If you need to use command because apt is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi051.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.023331", "end": "2020-11-09 18:04:03.777245", "rc": 0, "start": "2020-11-09 18:04:03.753914", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-09T18:04:04.164 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:04 +0000 (0:00:00.062) 0:00:55.890 ******* 2020-11-09T18:04:04.196 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:04 +0000 (0:00:00.031) 0:00:55.922 ******* 2020-11-09T18:04:04.341 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-11-09T18:04:04.343 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.013767", "end": "2020-11-09 18:04:04.757792", "rc": 0, "start": "2020-11-09 18:04:04.744025", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-11-09T18:04:04.379 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:04 +0000 (0:00:00.182) 0:00:56.104 ******* 2020-11-09T18:04:04.406 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2020-11-09T18:04:04.408 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:04.441 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:04 +0000 (0:00:00.062) 0:00:56.167 ******* 2020-11-09T18:04:04.468 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2020-11-09T18:04:04.470 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:04.503 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:04 +0000 (0:00:00.061) 0:00:56.229 ******* 2020-11-09T18:04:04.531 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-11-09T18:04:04.533 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:04.565 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:04 +0000 (0:00:00.061) 0:00:56.291 ******* 2020-11-09T18:04:04.591 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2020-11-09T18:04:04.593 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:04.625 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:04 +0000 (0:00:00.060) 0:00:56.351 ******* 2020-11-09T18:04:04.652 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2020-11-09T18:04:04.654 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:04.687 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:04 +0000 (0:00:00.061) 0:00:56.412 ******* 2020-11-09T18:04:04.713 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2020-11-09T18:04:04.715 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:04.747 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:04 +0000 (0:00:00.060) 0:00:56.473 ******* 2020-11-09T18:04:04.774 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-11-09T18:04:04.775 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:04.815 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:04 +0000 (0:00:00.067) 0:00:56.541 ******* 2020-11-09T18:04:05.085 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2020-11-09T18:04:05.087 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.144408", "end": "2020-11-09 18:04:05.500382", "rc": 0, "start": "2020-11-09 18:04:05.355974", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-09T18:04:05.119 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:05 +0000 (0:00:00.304) 0:00:56.845 ******* 2020-11-09T18:04:05.146 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-11-09T18:04:05.148 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:05.197 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:05 +0000 (0:00:00.076) 0:00:56.922 ******* 2020-11-09T18:04:05.352 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2020-11-09T18:04:05.354 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.025036", "end": "2020-11-09 18:04:05.768183", "rc": 0, "start": "2020-11-09 18:04:05.743147", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-09T18:04:05.388 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:05 +0000 (0:00:00.191) 0:00:57.114 ******* 2020-11-09T18:04:07.374 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2020-11-09T18:04:07.376 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1604944975, "cache_updated": false, "changed": false} 2020-11-09T18:04:07.408 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:07 +0000 (0:00:02.020) 0:00:59.134 ******* 2020-11-09T18:04:08.252 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1604944981, "cache_updated": false, "changed": false} 2020-11-09T18:04:08.287 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:08 +0000 (0:00:00.878) 0:01:00.013 ******* 2020-11-09T18:04:08.294 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2020-11-09T18:04:08.296 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"cache_update_time": 1604944975, "cache_updated": false, "changed": false} 2020-11-09T18:04:08.331 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:08 +0000 (0:00:00.043) 0:01:00.056 ******* 2020-11-09T18:04:08.359 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2020-11-09T18:04:08.361 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:08.393 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:08 +0000 (0:00:00.062) 0:01:00.119 ******* 2020-11-09T18:04:08.960 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2020-11-09T18:04:08.962 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2020-11-09T18:04:08.995 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:08 +0000 (0:00:00.601) 0:01:00.721 ******* 2020-11-09T18:04:09.186 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2020-11-09T18:04:09.188 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"cache_update_time": 1604944981, "cache_updated": false, "changed": false} 2020-11-09T18:04:09.221 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:09 +0000 (0:00:00.226) 0:01:00.947 ******* 2020-11-09T18:04:09.249 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2020-11-09T18:04:09.251 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:09.284 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:09 +0000 (0:00:00.062) 0:01:01.010 ******* 2020-11-09T18:04:09.824 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2020-11-09T18:04:09.826 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2020-11-09T18:04:09.875 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:09 +0000 (0:00:00.590) 0:01:01.600 ******* 2020-11-09T18:04:09.882 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2020-11-09T18:04:09.884 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"cache_update_time": 1604944975, "cache_updated": false, "changed": false} 2020-11-09T18:04:09.919 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:09 +0000 (0:00:00.044) 0:01:01.645 ******* 2020-11-09T18:04:10.250 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-11-09T18:04:10.252 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2020-11-09T18:04:10.543 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2020-11-09T18:04:10.741 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2020-11-09T18:04:10.743 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"cache_update_time": 1604944981, "cache_updated": false, "changed": false} 2020-11-09T18:04:10.777 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:10 +0000 (0:00:00.857) 0:01:02.503 ******* 2020-11-09T18:04:10.818 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-11-09T18:04:10.820 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2020-11-09T18:04:10.859 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:10 +0000 (0:00:00.082) 0:01:02.585 ******* 2020-11-09T18:04:10.982 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2020-11-09T18:04:10.984 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2020-11-09T18:04:11.026 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:11 +0000 (0:00:00.166) 0:01:02.752 ******* 2020-11-09T18:04:11.119 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-11-09T18:04:11.121 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2020-11-09T18:04:11.348 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-11-09T18:04:11.350 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 12725, "state": "file", "uid": 0} 2020-11-09T18:04:11.445 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-11-09T18:04:11.447 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2020-11-09T18:04:11.639 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-11-09T18:04:11.641 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2020-11-09T18:04:11.758 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-11-09T18:04:11.759 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2020-11-09T18:04:11.799 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:11 +0000 (0:00:00.772) 0:01:03.524 ******* 2020-11-09T18:04:11.926 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2020-11-09T18:04:11.927 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** ok: [smithi051.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2020-11-09T18:04:11.962 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:11 +0000 (0:00:00.163) 0:01:03.688 ******* 2020-11-09T18:04:11.995 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:11 +0000 (0:00:00.032) 0:01:03.721 ******* 2020-11-09T18:04:12.003 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2020-11-09T18:04:12.003 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2020-11-09T18:04:12.040 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:12 +0000 (0:00:00.044) 0:01:03.766 ******* 2020-11-09T18:04:12.079 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2020-11-09T18:04:12.081 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:12.113 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:12 +0000 (0:00:00.072) 0:01:03.839 ******* 2020-11-09T18:04:12.150 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* skipping: [smithi051.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-09T18:04:12.185 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:12 +0000 (0:00:00.071) 0:01:03.910 ******* 2020-11-09T18:04:12.223 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2020-11-09T18:04:12.225 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi051.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-09T18:04:12.235 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-09T18:04:12.246 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-09T18:04:12.281 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:12 +0000 (0:00:00.096) 0:01:04.007 ******* 2020-11-09T18:04:12.309 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-11-09T18:04:12.311 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 12725, "state": "file", "uid": 0} TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:12.347 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:12 +0000 (0:00:00.066) 0:01:04.073 ******* 2020-11-09T18:04:12.606 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** ok: [smithi078.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2020-11-09T18:04:12.901 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2020-11-09T18:04:12.941 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:12 +0000 (0:00:00.593) 0:01:04.667 ******* 2020-11-09T18:04:12.979 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2020-11-09T18:04:12.981 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2020-11-09T18:04:13.016 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:13 +0000 (0:00:00.075) 0:01:04.742 ******* 2020-11-09T18:04:13.043 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2020-11-09T18:04:13.045 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:13.077 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:13 +0000 (0:00:00.060) 0:01:04.803 ******* 2020-11-09T18:04:13.104 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2020-11-09T18:04:13.106 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-09T18:04:13.147 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:13 +0000 (0:00:00.069) 0:01:04.872 ******* 2020-11-09T18:04:13.185 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** skipping: [smithi078.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-09T18:04:13.196 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi078.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-09T18:04:13.208 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-09T18:04:13.243 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:13 +0000 (0:00:00.096) 0:01:04.969 ******* 2020-11-09T18:04:13.260 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2020-11-09T18:04:13.262 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"cache_update_time": 1604944975, "cache_updated": false, "changed": false} 2020-11-09T18:04:13.297 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:13 +0000 (0:00:00.054) 0:01:05.023 ******* 2020-11-09T18:04:13.304 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2020-11-09T18:04:13.306 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:13.340 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:13 +0000 (0:00:00.043) 0:01:05.066 ******* 2020-11-09T18:04:13.610 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2020-11-09T18:04:13.612 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2020-11-09T18:04:13.646 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:13 +0000 (0:00:00.305) 0:01:05.371 ******* 2020-11-09T18:04:13.969 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2020-11-09T18:04:13.971 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2020-11-09T18:04:14.004 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:14 +0000 (0:00:00.358) 0:01:05.730 ******* 2020-11-09T18:04:14.147 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-11-09T18:04:14.149 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2020-11-09T18:04:14.242 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2020-11-09T18:04:14.244 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"cache_update_time": 1604944981, "cache_updated": false, "changed": false} 2020-11-09T18:04:14.279 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:14 +0000 (0:00:00.274) 0:01:06.005 ******* 2020-11-09T18:04:14.286 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-11-09T18:04:14.288 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2020-11-09T18:04:14.330 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:14 +0000 (0:00:00.050) 0:01:06.055 ******* 2020-11-09T18:04:14.359 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2020-11-09T18:04:14.361 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:14.404 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:14 +0000 (0:00:00.074) 0:01:06.129 ******* 2020-11-09T18:04:14.435 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** skipping: [smithi051.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-09T18:04:14.472 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:14 +0000 (0:00:00.068) 0:01:06.198 ******* 2020-11-09T18:04:14.501 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2020-11-09T18:04:14.503 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:14.535 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:14 +0000 (0:00:00.062) 0:01:06.261 ******* 2020-11-09T18:04:14.616 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2020-11-09T18:04:14.618 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2020-11-09T18:04:14.663 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:14 +0000 (0:00:00.127) 0:01:06.388 ******* 2020-11-09T18:04:14.863 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2020-11-09T18:04:14.865 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2020-11-09T18:04:14.904 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:14 +0000 (0:00:00.241) 0:01:06.630 ******* 2020-11-09T18:04:14.988 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2020-11-09T18:04:14.990 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2020-11-09T18:04:15.023 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:15 +0000 (0:00:00.118) 0:01:06.749 ******* 2020-11-09T18:04:15.185 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-11-09T18:04:15.187 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2020-11-09T18:04:15.303 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2020-11-09T18:04:15.356 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:15 +0000 (0:00:00.332) 0:01:07.081 ******* 2020-11-09T18:04:15.384 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2020-11-09T18:04:15.386 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:15.431 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:15 +0000 (0:00:00.075) 0:01:07.157 ******* 2020-11-09T18:04:15.459 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** skipping: [smithi078.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-09T18:04:15.493 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:15 +0000 (0:00:00.062) 0:01:07.219 ******* 2020-11-09T18:04:15.522 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2020-11-09T18:04:15.530 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:15.570 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:15 +0000 (0:00:00.076) 0:01:07.296 ******* 2020-11-09T18:04:15.580 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2020-11-09T18:04:15.581 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Mon 2020-11-09 18:01:50 UTC", "ActiveEnterTimestampMonotonic": "17329943", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "basic.target remote-fs.target nss-lookup.target local-fs.target systemd-journald.socket system.slice sysinit.target -.mount time-sync.target network.target systemd-tmpfiles-setup.service var-run.mount", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2020-11-09 18:01:50 UTC", "AssertTimestampMonotonic": "17329223", "Before": "multi-user.target getty@tty1.service plymouth-quit.service shutdown.target xdm.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2020-11-09 18:01:50 UTC", "ConditionTimestampMonotonic": "17329222", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nagios-nrpe-ser 2020-11-09T18:04:15.583 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1148", "ExecMainStartTimestamp": "Mon 2020-11-09 18:01:50 UTC", "ExecMainStartTimestampMonotonic": "17329916", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2020-11-09 18:01:50 UTC", "InactiveExitTimestampMonotonic": "17329943", "InvocationID": "1461da2775a340aaa88575f1a703f32a", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "i 2020-11-09T18:04:15.584 INFO:teuthology.task.ansible.out:nfinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128055", "LimitNPROCSoft": "128055", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128055", "LimitSIGPENDINGSoft": "128055", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1148", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no" 2020-11-09T18:04:15.586 INFO:teuthology.task.ansible.out:, "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2020-11-09 18:01:50 UTC", "StateChangeTimestampMonotonic": "17329943", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2020-11-09 18:01:50 UTC", "WatchdogTimestampMonotonic": "17329942", "WatchdogUSec": "0"}} 2020-11-09T18:04:15.622 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:15 +0000 (0:00:00.051) 0:01:07.348 ******* 2020-11-09T18:04:15.654 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:15.687 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:15 +0000 (0:00:00.064) 0:01:07.412 ******* 2020-11-09T18:04:15.713 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2020-11-09T18:04:15.717 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:15.749 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:15 +0000 (0:00:00.061) 0:01:07.474 ******* 2020-11-09T18:04:15.790 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:15.825 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:15 +0000 (0:00:00.077) 0:01:07.551 ******* 2020-11-09T18:04:15.854 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2020-11-09T18:04:15.856 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:15.900 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:15 +0000 (0:00:00.074) 0:01:07.626 ******* 2020-11-09T18:04:15.909 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2020-11-09T18:04:15.910 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2020-11-09T18:04:15.945 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:15 +0000 (0:00:00.044) 0:01:07.671 ******* 2020-11-09T18:04:15.952 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2020-11-09T18:04:15.954 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:15.988 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:15 +0000 (0:00:00.042) 0:01:07.714 ******* 2020-11-09T18:04:16.018 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2020-11-09T18:04:16.020 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:16.053 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:16 +0000 (0:00:00.065) 0:01:07.779 ******* 2020-11-09T18:04:16.081 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2020-11-09T18:04:16.083 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:16.116 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:16 +0000 (0:00:00.062) 0:01:07.841 ******* 2020-11-09T18:04:16.144 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2020-11-09T18:04:16.146 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [common : Make sure nagios nrpe service is running.] ********************** 2020-11-09T18:04:16.148 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Mon 2020-11-09 18:02:18 UTC", "ActiveEnterTimestampMonotonic": "17646483", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "system.slice time-sync.target sysinit.target systemd-journald.socket remote-fs.target local-fs.target nss-lookup.target basic.target -.mount var-run.mount network.target systemd-tmpfiles-setup.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2020-11-09 18:02:18 UTC", "AssertTimestampMonotonic": "17645365", "Before": "getty@tty1.service xdm.service shutdown.target plymouth-quit.service multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2020-11-09 18:02:18 UTC", "ConditionTimestampMonotonic": "17645364", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nagios-nrpe-ser 2020-11-09T18:04:16.149 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1455", "ExecMainStartTimestamp": "Mon 2020-11-09 18:02:18 UTC", "ExecMainStartTimestampMonotonic": "17646449", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2020-11-09 18:02:18 UTC", "InactiveExitTimestampMonotonic": "17646483", "InvocationID": "6e21bb8899604f33944dbb3c73721585", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "i 2020-11-09T18:04:16.151 INFO:teuthology.task.ansible.out:nfinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128030", "LimitNPROCSoft": "128030", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128030", "LimitSIGPENDINGSoft": "128030", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1455", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit.target", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no" 2020-11-09T18:04:16.153 INFO:teuthology.task.ansible.out:, "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2020-11-09 18:02:18 UTC", "StateChangeTimestampMonotonic": "17646483", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2020-11-09 18:02:18 UTC", "WatchdogTimestampMonotonic": "17646482", "WatchdogUSec": "0"}} 2020-11-09T18:04:16.183 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:16 +0000 (0:00:00.066) 0:01:07.908 ******* 2020-11-09T18:04:16.215 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:16 +0000 (0:00:00.032) 0:01:07.941 ******* 2020-11-09T18:04:16.221 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2020-11-09T18:04:16.223 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:16.257 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:16 +0000 (0:00:00.042) 0:01:07.983 ******* 2020-11-09T18:04:16.264 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2020-11-09T18:04:16.265 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:16.300 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:16 +0000 (0:00:00.042) 0:01:08.025 ******* 2020-11-09T18:04:16.307 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:16.346 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:16 +0000 (0:00:00.046) 0:01:08.072 ******* 2020-11-09T18:04:16.353 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:16.389 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:16 +0000 (0:00:00.042) 0:01:08.115 ******* 2020-11-09T18:04:16.396 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** ok: [smithi051.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2020-11-09T18:04:16.432 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:16 +0000 (0:00:00.043) 0:01:08.158 ******* 2020-11-09T18:04:16.440 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2020-11-09T18:04:16.441 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:16.475 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:16 +0000 (0:00:00.042) 0:01:08.201 ******* 2020-11-09T18:04:16.482 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2020-11-09T18:04:16.484 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2020-11-09T18:04:16.519 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:16 +0000 (0:00:00.044) 0:01:08.245 ******* 2020-11-09T18:04:16.527 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2020-11-09T18:04:16.529 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:16.563 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:16 +0000 (0:00:00.043) 0:01:08.289 ******* 2020-11-09T18:04:16.570 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2020-11-09T18:04:16.572 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2020-11-09T18:04:16.606 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:16 +0000 (0:00:00.042) 0:01:08.332 ******* 2020-11-09T18:04:16.613 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2020-11-09T18:04:16.615 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:16.650 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:16 +0000 (0:00:00.043) 0:01:08.376 ******* 2020-11-09T18:04:16.657 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2020-11-09T18:04:16.658 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-11-09T18:04:16.694 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:16 +0000 (0:00:00.044) 0:01:08.420 ******* 2020-11-09T18:04:16.701 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2020-11-09T18:04:16.702 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:16.739 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:16 +0000 (0:00:00.044) 0:01:08.465 ******* 2020-11-09T18:04:16.767 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2020-11-09T18:04:16.768 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:16.802 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:16 +0000 (0:00:00.063) 0:01:08.528 ******* 2020-11-09T18:04:16.829 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** ok: [smithi051.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-11-09T18:04:16.831 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:16.863 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:16 +0000 (0:00:00.061) 0:01:08.589 ******* 2020-11-09T18:04:16.901 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:16 +0000 (0:00:00.037) 0:01:08.627 ******* 2020-11-09T18:04:16.928 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2020-11-09T18:04:16.930 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:16.963 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:16 +0000 (0:00:00.061) 0:01:08.689 ******* 2020-11-09T18:04:16.997 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:16.999 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* ok: [smithi051.front.sepia.ceph.com] => {"changed": false, "gid": 1234, "name": "ubuntu", "state": "present", "system": false} 2020-11-09T18:04:17.138 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:17 +0000 (0:00:00.175) 0:01:08.864 ******* 2020-11-09T18:04:17.171 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:17 +0000 (0:00:00.032) 0:01:08.897 ******* 2020-11-09T18:04:17.178 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2020-11-09T18:04:17.179 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:17.213 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:17 +0000 (0:00:00.042) 0:01:08.939 ******* 2020-11-09T18:04:17.220 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2020-11-09T18:04:17.222 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2020-11-09T18:04:17.255 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:17 +0000 (0:00:00.042) 0:01:08.981 ******* 2020-11-09T18:04:17.262 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-11-09T18:04:17.263 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:17.297 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:17 +0000 (0:00:00.041) 0:01:09.023 ******* 2020-11-09T18:04:17.304 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2020-11-09T18:04:17.305 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2020-11-09T18:04:17.341 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:17 +0000 (0:00:00.043) 0:01:09.067 ******* 2020-11-09T18:04:17.381 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2020-11-09T18:04:17.382 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2020-11-09T18:04:17.419 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:17 +0000 (0:00:00.078) 0:01:09.145 ******* 2020-11-09T18:04:17.447 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2020-11-09T18:04:17.449 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2020-11-09T18:04:17.481 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:17 +0000 (0:00:00.062) 0:01:09.207 ******* 2020-11-09T18:04:17.490 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ ok: [smithi078.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-11-09T18:04:17.526 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:17 +0000 (0:00:00.044) 0:01:09.252 ******* 2020-11-09T18:04:17.652 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2020-11-09T18:04:17.654 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-11-09T18:04:17.687 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:17 +0000 (0:00:00.160) 0:01:09.413 ******* 2020-11-09T18:04:17.819 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2020-11-09T18:04:17.820 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "gid": 1234, "name": "ubuntu", "state": "present", "system": false} 2020-11-09T18:04:17.853 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:17 +0000 (0:00:00.165) 0:01:09.579 ******* 2020-11-09T18:04:17.869 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2020-11-09T18:04:17.871 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2020-11-09T18:04:17.903 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:17 +0000 (0:00:00.050) 0:01:09.629 ******* 2020-11-09T18:04:17.910 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2020-11-09T18:04:17.912 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:17.954 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:17 +0000 (0:00:00.050) 0:01:09.680 ******* 2020-11-09T18:04:17.963 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-11-09T18:04:17.965 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi051"}, "changed": false} 2020-11-09T18:04:18.012 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:18 +0000 (0:00:00.057) 0:01:09.737 ******* 2020-11-09T18:04:18.021 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-11-09T18:04:18.023 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:18.057 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:18 +0000 (0:00:00.045) 0:01:09.782 ******* 2020-11-09T18:04:18.212 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2020-11-09T18:04:18.214 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2020-11-09T18:04:18.246 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:18 +0000 (0:00:00.188) 0:01:09.971 ******* 2020-11-09T18:04:18.623 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2020-11-09T18:04:18.659 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:18 +0000 (0:00:00.412) 0:01:10.384 ******* 2020-11-09T18:04:18.686 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-11-09T18:04:18.687 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi078"}, "changed": false} 2020-11-09T18:04:18.721 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:18 +0000 (0:00:00.062) 0:01:10.447 ******* 2020-11-09T18:04:18.877 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2020-11-09T18:04:18.879 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi051", "ansible_hostname": "smithi051", "ansible_nodename": "smithi051"}, "changed": false, "name": "smithi051"} 2020-11-09T18:04:18.913 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:18 +0000 (0:00:00.192) 0:01:10.639 ******* 2020-11-09T18:04:19.218 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi078", "ansible_hostname": "smithi078", "ansible_nodename": "smithi078"}, "changed": false, "name": "smithi078"} 2020-11-09T18:04:19.256 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:19 +0000 (0:00:00.342) 0:01:10.982 ******* 2020-11-09T18:04:19.262 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2020-11-09T18:04:19.264 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2020-11-09T18:04:19.301 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:19 +0000 (0:00:00.044) 0:01:11.026 ******* 2020-11-09T18:04:19.581 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2020-11-09T18:04:19.621 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:19 +0000 (0:00:00.319) 0:01:11.346 ******* 2020-11-09T18:04:19.627 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2020-11-09T18:04:19.629 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2020-11-09T18:04:19.663 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:19 +0000 (0:00:00.042) 0:01:11.389 ******* 2020-11-09T18:04:19.935 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2020-11-09T18:04:19.970 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:19 +0000 (0:00:00.306) 0:01:11.696 ******* 2020-11-09T18:04:20.199 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2020-11-09T18:04:20.201 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"attempts": 1, "changed": false, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-11-09T18:04:20.237 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:20 +0000 (0:00:00.265) 0:01:11.962 ******* 2020-11-09T18:04:20.265 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2020-11-09T18:04:20.267 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:20.301 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:20 +0000 (0:00:00.064) 0:01:12.027 ******* 2020-11-09T18:04:20.340 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:20.374 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:20 +0000 (0:00:00.073) 0:01:12.100 ******* 2020-11-09T18:04:20.401 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:20.435 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:20 +0000 (0:00:00.060) 0:01:12.161 ******* 2020-11-09T18:04:20.463 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-11-09T18:04:20.464 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:20.498 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:20 +0000 (0:00:00.062) 0:01:12.224 ******* 2020-11-09T18:04:20.516 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2020-11-09T18:04:20.518 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"attempts": 1, "changed": false, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-11-09T18:04:20.552 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:20 +0000 (0:00:00.054) 0:01:12.278 ******* 2020-11-09T18:04:20.558 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-11-09T18:04:20.560 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:20.594 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:20 +0000 (0:00:00.041) 0:01:12.320 ******* 2020-11-09T18:04:20.601 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2020-11-09T18:04:20.603 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:20.637 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:20 +0000 (0:00:00.042) 0:01:12.362 ******* 2020-11-09T18:04:20.644 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-11-09T18:04:20.645 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:20.678 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:20 +0000 (0:00:00.041) 0:01:12.404 ******* 2020-11-09T18:04:20.684 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-11-09T18:04:20.685 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:20.723 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:20 +0000 (0:00:00.044) 0:01:12.448 ******* 2020-11-09T18:04:20.729 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-11-09T18:04:20.731 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:20.765 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:20 +0000 (0:00:00.042) 0:01:12.491 ******* 2020-11-09T18:04:20.773 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-11-09T18:04:20.775 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:20.809 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:20 +0000 (0:00:00.043) 0:01:12.535 ******* 2020-11-09T18:04:20.816 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2020-11-09T18:04:20.818 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:20.852 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:20 +0000 (0:00:00.042) 0:01:12.578 ******* 2020-11-09T18:04:20.859 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-11-09T18:04:20.860 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:20.893 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:20 +0000 (0:00:00.040) 0:01:12.619 ******* 2020-11-09T18:04:20.899 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-11-09T18:04:20.935 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:20 +0000 (0:00:00.041) 0:01:12.661 ******* 2020-11-09T18:04:20.941 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-11-09T18:04:20.943 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:20.976 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:20 +0000 (0:00:00.041) 0:01:12.702 ******* 2020-11-09T18:04:20.982 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-11-09T18:04:21.018 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:21 +0000 (0:00:00.042) 0:01:12.744 ******* 2020-11-09T18:04:21.025 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-11-09T18:04:21.026 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:21.060 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:21 +0000 (0:00:00.042) 0:01:12.786 ******* 2020-11-09T18:04:21.069 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-11-09T18:04:21.105 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:21 +0000 (0:00:00.044) 0:01:12.831 ******* 2020-11-09T18:04:21.112 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-11-09T18:04:21.113 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:21.148 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:21 +0000 (0:00:00.042) 0:01:12.874 ******* 2020-11-09T18:04:21.155 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2020-11-09T18:04:21.156 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:21.191 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:21 +0000 (0:00:00.043) 0:01:12.917 ******* 2020-11-09T18:04:21.199 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:21.235 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:21 +0000 (0:00:00.043) 0:01:12.960 ******* 2020-11-09T18:04:21.241 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* skipping: [smithi051.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi051.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-11-09T18:04:21.276 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:21 +0000 (0:00:00.041) 0:01:13.002 ******* 2020-11-09T18:04:21.284 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-11-09T18:04:21.322 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:21 +0000 (0:00:00.045) 0:01:13.048 ******* 2020-11-09T18:04:21.329 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2020-11-09T18:04:21.331 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:21.365 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:21 +0000 (0:00:00.042) 0:01:13.090 ******* 2020-11-09T18:04:21.373 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-11-09T18:04:21.407 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:21 +0000 (0:00:00.042) 0:01:13.132 ******* 2020-11-09T18:04:21.413 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-09T18:04:21.415 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:21.449 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:21 +0000 (0:00:00.042) 0:01:13.174 ******* 2020-11-09T18:04:21.455 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-11-09T18:04:21.491 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:21 +0000 (0:00:00.042) 0:01:13.217 ******* 2020-11-09T18:04:21.498 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2020-11-09T18:04:21.500 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:21.533 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:21 +0000 (0:00:00.042) 0:01:13.259 ******* 2020-11-09T18:04:21.540 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:21.574 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:21 +0000 (0:00:00.041) 0:01:13.300 ******* 2020-11-09T18:04:21.580 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2020-11-09T18:04:21.582 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:21.618 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:21 +0000 (0:00:00.043) 0:01:13.343 ******* 2020-11-09T18:04:21.626 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* skipping: [smithi078.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-11-09T18:04:21.628 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-11-09T18:04:21.661 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:21 +0000 (0:00:00.043) 0:01:13.387 ******* 2020-11-09T18:04:21.669 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2020-11-09T18:04:21.670 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:21.703 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:21 +0000 (0:00:00.041) 0:01:13.428 ******* 2020-11-09T18:04:21.709 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2020-11-09T18:04:21.711 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:21.745 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:21 +0000 (0:00:00.042) 0:01:13.471 ******* 2020-11-09T18:04:21.751 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-11-09T18:04:21.753 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:21.787 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:21 +0000 (0:00:00.042) 0:01:13.513 ******* 2020-11-09T18:04:21.795 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:21.843 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:21 +0000 (0:00:00.054) 0:01:13.568 ******* 2020-11-09T18:04:21.849 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-11-09T18:04:21.851 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:21.884 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:21 +0000 (0:00:00.041) 0:01:13.609 ******* 2020-11-09T18:04:21.891 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2020-11-09T18:04:21.892 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:21.928 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:21 +0000 (0:00:00.044) 0:01:13.654 ******* 2020-11-09T18:04:21.935 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-11-09T18:04:21.937 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:21.970 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:21 +0000 (0:00:00.041) 0:01:13.696 ******* 2020-11-09T18:04:21.978 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2020-11-09T18:04:21.981 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:22.013 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:22 +0000 (0:00:00.042) 0:01:13.739 ******* 2020-11-09T18:04:22.020 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-11-09T18:04:22.021 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:22.057 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:22 +0000 (0:00:00.044) 0:01:13.783 ******* 2020-11-09T18:04:22.065 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:22.099 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:22 +0000 (0:00:00.041) 0:01:13.825 ******* 2020-11-09T18:04:22.106 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-11-09T18:04:22.108 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:22.144 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:22 +0000 (0:00:00.044) 0:01:13.869 ******* 2020-11-09T18:04:22.150 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-11-09T18:04:22.152 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:22.186 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:22 +0000 (0:00:00.042) 0:01:13.911 ******* 2020-11-09T18:04:22.192 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-11-09T18:04:22.194 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:22.228 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:22 +0000 (0:00:00.042) 0:01:13.954 ******* 2020-11-09T18:04:22.235 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-11-09T18:04:22.236 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:22.270 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:22 +0000 (0:00:00.041) 0:01:13.995 ******* 2020-11-09T18:04:22.277 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:22.311 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:22 +0000 (0:00:00.041) 0:01:14.037 ******* 2020-11-09T18:04:22.318 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-11-09T18:04:22.320 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:22.353 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:22 +0000 (0:00:00.041) 0:01:14.079 ******* 2020-11-09T18:04:22.359 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2020-11-09T18:04:22.361 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:22.394 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:22 +0000 (0:00:00.040) 0:01:14.120 ******* 2020-11-09T18:04:22.400 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-11-09T18:04:22.402 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:22.435 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:22 +0000 (0:00:00.040) 0:01:14.161 ******* 2020-11-09T18:04:22.441 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2020-11-09T18:04:22.442 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:22.477 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:22 +0000 (0:00:00.042) 0:01:14.203 ******* 2020-11-09T18:04:22.484 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-11-09T18:04:22.487 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:22.522 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:22 +0000 (0:00:00.044) 0:01:14.248 ******* 2020-11-09T18:04:22.529 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2020-11-09T18:04:22.530 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:22.565 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:22 +0000 (0:00:00.042) 0:01:14.291 ******* 2020-11-09T18:04:22.573 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-11-09T18:04:22.575 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:22.607 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:22 +0000 (0:00:00.042) 0:01:14.333 ******* 2020-11-09T18:04:22.614 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:22.649 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:22 +0000 (0:00:00.041) 0:01:14.375 ******* 2020-11-09T18:04:22.656 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-11-09T18:04:22.657 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:22.692 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:22 +0000 (0:00:00.042) 0:01:14.417 ******* 2020-11-09T18:04:22.699 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2020-11-09T18:04:22.701 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:22.735 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:22 +0000 (0:00:00.042) 0:01:14.460 ******* 2020-11-09T18:04:22.741 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2020-11-09T18:04:22.743 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:22.776 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:22 +0000 (0:00:00.041) 0:01:14.502 ******* 2020-11-09T18:04:22.783 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2020-11-09T18:04:22.784 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:22.817 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:22 +0000 (0:00:00.041) 0:01:14.543 ******* 2020-11-09T18:04:22.824 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2020-11-09T18:04:22.826 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:22.859 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:22 +0000 (0:00:00.041) 0:01:14.585 ******* 2020-11-09T18:04:22.867 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2020-11-09T18:04:22.868 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:22.902 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:22 +0000 (0:00:00.042) 0:01:14.627 ******* 2020-11-09T18:04:22.909 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2020-11-09T18:04:22.911 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:22.951 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:22 +0000 (0:00:00.048) 0:01:14.676 ******* 2020-11-09T18:04:22.958 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2020-11-09T18:04:22.960 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:22.997 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:22 +0000 (0:00:00.044) 0:01:14.721 ******* 2020-11-09T18:04:23.006 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2020-11-09T18:04:23.008 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:23.058 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:23 +0000 (0:00:00.062) 0:01:14.783 ******* 2020-11-09T18:04:23.088 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:23.137 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:23 +0000 (0:00:00.079) 0:01:14.863 ******* 2020-11-09T18:04:23.157 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2020-11-09T18:04:23.158 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005139", "end": "2020-11-09 18:04:22.801386", "rc": 0, "start": "2020-11-09 18:04:22.796247", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-09T18:04:23.214 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:23 +0000 (0:00:00.076) 0:01:14.940 ******* 2020-11-09T18:04:23.222 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2020-11-09T18:04:23.224 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:23.258 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:23 +0000 (0:00:00.044) 0:01:14.984 ******* 2020-11-09T18:04:23.265 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2020-11-09T18:04:23.303 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:23 +0000 (0:00:00.044) 0:01:15.029 ******* 2020-11-09T18:04:23.311 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2020-11-09T18:04:23.312 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:23.347 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:23 +0000 (0:00:00.044) 0:01:15.073 ******* 2020-11-09T18:04:23.376 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2020-11-09T18:04:23.378 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:23.411 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:23 +0000 (0:00:00.063) 0:01:15.136 ******* 2020-11-09T18:04:23.553 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2020-11-09T18:04:23.555 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005111", "end": "2020-11-09 18:04:23.961325", "rc": 0, "start": "2020-11-09 18:04:23.956214", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-09T18:04:23.597 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:23 +0000 (0:00:00.186) 0:01:15.323 ******* 2020-11-09T18:04:23.625 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2020-11-09T18:04:23.662 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:23 +0000 (0:00:00.064) 0:01:15.387 ******* 2020-11-09T18:04:23.671 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2020-11-09T18:04:23.672 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2020-11-09T18:04:23.707 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:23 +0000 (0:00:00.045) 0:01:15.433 ******* 2020-11-09T18:04:23.734 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2020-11-09T18:04:23.736 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:23.770 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:23 +0000 (0:00:00.061) 0:01:15.494 ******* 2020-11-09T18:04:24.019 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2020-11-09T18:04:24.021 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2020-11-09T18:04:24.060 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:24 +0000 (0:00:00.290) 0:01:15.785 ******* 2020-11-09T18:04:24.087 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:24.125 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:24 +0000 (0:00:00.065) 0:01:15.851 ******* 2020-11-09T18:04:25.070 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2020-11-09T18:04:25.072 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2020-11-09T18:04:25.289 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2020-11-09T18:04:26.018 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2020-11-09T18:04:26.055 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:26 +0000 (0:00:01.930) 0:01:17.781 ******* 2020-11-09T18:04:26.220 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2020-11-09T18:04:26.256 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:26 +0000 (0:00:00.201) 0:01:17.982 ******* 2020-11-09T18:04:26.977 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* ok: [smithi051.front.sepia.ceph.com] => {"cache_update_time": 1604944975, "cache_updated": false, "changed": false} 2020-11-09T18:04:27.014 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:27 +0000 (0:00:00.757) 0:01:18.740 ******* 2020-11-09T18:04:27.043 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-11-09T18:04:27.079 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:27 +0000 (0:00:00.065) 0:01:18.805 ******* 2020-11-09T18:04:27.173 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2020-11-09T18:04:27.175 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"cache_update_time": 1604944981, "cache_updated": false, "changed": false} 2020-11-09T18:04:27.211 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:27 +0000 (0:00:00.131) 0:01:18.936 ******* 2020-11-09T18:04:27.244 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-11-09T18:04:27.245 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** [WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running sudo changed: [smithi051.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.027846", "end": "2020-11-09 18:04:26.887389", "rc": 0, "start": "2020-11-09 18:04:26.859543", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-09T18:04:27.278 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:27 +0000 (0:00:00.067) 0:01:19.004 ******* 2020-11-09T18:04:27.316 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:27 +0000 (0:00:00.038) 0:01:19.042 ******* 2020-11-09T18:04:27.491 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.032788", "end": "2020-11-09 18:04:27.889752", "rc": 0, "start": "2020-11-09 18:04:27.856964", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-09T18:04:27.525 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:27 +0000 (0:00:00.208) 0:01:19.251 ******* 2020-11-09T18:04:30.278 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2020-11-09T18:04:30.280 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"cache_update_time": 1604944975, "cache_updated": false, "changed": false} 2020-11-09T18:04:30.313 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:30 +0000 (0:00:02.787) 0:01:22.039 ******* 2020-11-09T18:04:30.404 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"cache_update_time": 1604944981, "cache_updated": false, "changed": false} 2020-11-09T18:04:30.441 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:30 +0000 (0:00:00.128) 0:01:22.167 ******* 2020-11-09T18:04:39.860 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** changed: [smithi078.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 2 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 2020-11-09T18:04:39.862 INFO:teuthology.task.ansible.out: 95%\r(Reading database ... 100%\r(Reading database ... 102070 files and directories currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.3) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 2 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading d 2020-11-09T18:04:39.863 INFO:teuthology.task.ansible.out:atabase ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 102070 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.3) ..."]} 2020-11-09T18:04:39.894 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:39 +0000 (0:00:09.452) 0:01:31.620 ******* 2020-11-09T18:04:40.764 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-11-09T18:04:40.766 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"cache_update_time": 1604944981, "cache_updated": false, "changed": false} 2020-11-09T18:04:40.799 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:40 +0000 (0:00:00.905) 0:01:32.525 ******* 2020-11-09T18:04:40.826 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2020-11-09T18:04:40.828 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:40.861 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:40 +0000 (0:00:00.061) 0:01:32.586 ******* 2020-11-09T18:04:40.992 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2020-11-09T18:04:40.994 INFO:teuthology.task.ansible.out:changed: [smithi051.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 2 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 102070 files and directories 2020-11-09T18:04:40.996 INFO:teuthology.task.ansible.out: currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.3) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 2 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Readin 2020-11-09T18:04:40.998 INFO:teuthology.task.ansible.out:g database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 102070 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.3) ..."]} 2020-11-09T18:04:41.030 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:41 +0000 (0:00:00.168) 0:01:32.755 ******* 2020-11-09T18:04:41.951 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-11-09T18:04:41.953 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"cache_update_time": 1604944975, "cache_updated": false, "changed": false} 2020-11-09T18:04:41.995 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:41 +0000 (0:00:00.965) 0:01:33.721 ******* 2020-11-09T18:04:42.032 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2020-11-09T18:04:42.034 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:42.069 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:42 +0000 (0:00:00.074) 0:01:33.795 ******* 2020-11-09T18:04:49.572 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-11-09T18:04:49.573 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => {"cache_update_time": 1604944981, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 2 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.4 [2727 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.4 [924 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.33 [40.1 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.33 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.33 [5193 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.33 [873 kB]\nFetched 10.4 MB in 1s (9753 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 2020-11-09T18:04:49.575 INFO:teuthology.task.ansible.out: 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 101928 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.4_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.4_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nSetting up librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu 2020-11-09T18:04:49.577 INFO:teuthology.task.ansible.out:1.3) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 2 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.4 [2727 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.4 [924 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.33 [40.1 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.33 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.33 [5193 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.33 [873 kB]", "Fetched 10.4 MB in 1s (9753 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Rea 2020-11-09T18:04:49.578 INFO:teuthology.task.ansible.out:ding database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 101928 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.4_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.4_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.3) ..."]} 2020-11-09T18:04:49.613 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:49 +0000 (0:00:07.543) 0:01:41.338 ******* 2020-11-09T18:04:50.517 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2020-11-09T18:04:50.519 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"cache_update_time": 1604944981, "cache_updated": false, "changed": false} 2020-11-09T18:04:50.551 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:50 +0000 (0:00:00.938) 0:01:42.277 ******* 2020-11-09T18:04:50.632 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-11-09T18:04:50.634 INFO:teuthology.task.ansible.out:changed: [smithi051.front.sepia.ceph.com] => {"cache_update_time": 1604944975, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 2 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.4 [2727 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.4 [924 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.33 [40.1 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.33 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.33 [5193 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.33 [873 kB]\nFetched 10.4 MB in 1s (9776 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 2020-11-09T18:04:50.635 INFO:teuthology.task.ansible.out: 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 101928 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.4_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.4_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nSetting up librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu 2020-11-09T18:04:50.637 INFO:teuthology.task.ansible.out:1.3) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 2 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.4 [2727 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.4 [924 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.33 [40.1 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.33 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.33 [5193 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.33 [873 kB]", "Fetched 10.4 MB in 1s (9776 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Rea 2020-11-09T18:04:50.639 INFO:teuthology.task.ansible.out:ding database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 101928 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.4_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.4_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.3) ..."]} 2020-11-09T18:04:50.669 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:50 +0000 (0:00:00.118) 0:01:42.395 ******* 2020-11-09T18:04:50.687 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2020-11-09T18:04:50.689 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2020-11-09T18:04:50.724 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:50 +0000 (0:00:00.054) 0:01:42.450 ******* 2020-11-09T18:04:50.753 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:50.787 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:50 +0000 (0:00:00.062) 0:01:42.513 ******* 2020-11-09T18:04:50.929 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2020-11-09T18:04:50.931 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2020-11-09T18:04:50.964 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:50 +0000 (0:00:00.176) 0:01:42.689 ******* 2020-11-09T18:04:51.309 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2020-11-09T18:04:51.311 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2020-11-09T18:04:51.345 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:51 +0000 (0:00:00.380) 0:01:43.070 ******* 2020-11-09T18:04:51.488 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ ok: [smithi078.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1234, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-11-09T18:04:51.521 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:51 +0000 (0:00:00.176) 0:01:43.247 ******* 2020-11-09T18:04:51.582 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2020-11-09T18:04:51.585 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"cache_update_time": 1604944975, "cache_updated": false, "changed": false} 2020-11-09T18:04:51.620 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:51 +0000 (0:00:00.098) 0:01:43.345 ******* 2020-11-09T18:04:51.661 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2020-11-09T18:04:51.662 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-09T18:04:51.701 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:51 +0000 (0:00:00.081) 0:01:43.427 ******* 2020-11-09T18:04:51.750 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2020-11-09T18:04:51.753 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2020-11-09T18:04:51.787 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:51 +0000 (0:00:00.084) 0:01:43.512 ******* 2020-11-09T18:04:51.815 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2020-11-09T18:04:51.816 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:51.849 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:51 +0000 (0:00:00.062) 0:01:43.575 ******* 2020-11-09T18:04:51.983 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2020-11-09T18:04:51.985 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2020-11-09T18:04:52.033 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:52 +0000 (0:00:00.184) 0:01:43.759 ******* 2020-11-09T18:04:52.045 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-11-09T18:04:52.047 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Mon 2020-11-09 18:02:18 UTC", "ActiveEnterTimestampMonotonic": "18074160", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "nss-lookup.target systemd-tmpfiles-setup.service network.target sysinit.target basic.target systemd-journald.socket remote-fs.target -.mount system.slice", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2020-11-09 18:02:18 UTC", "AssertTimestampMonotonic": "17644431", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2020-11-09 18:02:18 UTC", "ConditionTimestampMonotonic": "17644431", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2020-11-09T18:04:52.049 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1565", "ExecMainStartTimestamp": "Mon 2020-11-09 18:02:18 UTC", "ExecMainStartTimestampMonotonic": "18074149", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2020-11-09 18:02:18 UTC", "InactiveExitTimestampMonotonic": "17645318", "InvocationID": "8ee8d22025b44c1a9075635738e141ec", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2020-11-09T18:04:52.050 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128030", "LimitNPROCSoft": "128030", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128030", "LimitSIGPENDINGSoft": "128030", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1565", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2020-11-09T18:04:52.052 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2020-11-09 18:02:18 UTC", "StateChangeTimestampMonotonic": "18074160", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2020-11-09 18:02:18 UTC", "WatchdogTimestampMonotonic": "18074159", "WatchdogUSec": "0"}} 2020-11-09T18:04:52.084 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:52 +0000 (0:00:00.051) 0:01:43.810 ******* 2020-11-09T18:04:52.111 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-11-09T18:04:52.112 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:52.145 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:52 +0000 (0:00:00.060) 0:01:43.870 ******* 2020-11-09T18:04:52.174 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-11-09T18:04:52.176 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:52.208 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:52 +0000 (0:00:00.063) 0:01:43.934 ******* 2020-11-09T18:04:52.236 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-11-09T18:04:52.237 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:52.270 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:52 +0000 (0:00:00.061) 0:01:43.996 ******* 2020-11-09T18:04:52.297 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-11-09T18:04:52.299 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:52.331 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:52 +0000 (0:00:00.061) 0:01:44.057 ******* 2020-11-09T18:04:52.359 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-11-09T18:04:52.362 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Upload /etc/fuse.conf.] *************************************** ok: [smithi051.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2020-11-09T18:04:52.396 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:52 +0000 (0:00:00.064) 0:01:44.121 ******* 2020-11-09T18:04:52.430 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:52 +0000 (0:00:00.034) 0:01:44.156 ******* 2020-11-09T18:04:52.457 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-11-09T18:04:52.458 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:52.498 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:52 +0000 (0:00:00.068) 0:01:44.224 ******* 2020-11-09T18:04:52.526 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-11-09T18:04:52.530 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:52.563 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:52 +0000 (0:00:00.064) 0:01:44.289 ******* 2020-11-09T18:04:52.570 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2020-11-09T18:04:52.572 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1234, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-11-09T18:04:52.605 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:52 +0000 (0:00:00.042) 0:01:44.331 ******* 2020-11-09T18:04:52.613 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-11-09T18:04:52.615 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:52.648 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:52 +0000 (0:00:00.043) 0:01:44.374 ******* 2020-11-09T18:04:52.677 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-09T18:04:52.679 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:52.714 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:52 +0000 (0:00:00.064) 0:01:44.439 ******* 2020-11-09T18:04:52.742 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2020-11-09T18:04:52.744 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} TASK [testnode : Remove packages] ********************************************** skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:52.779 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:52 +0000 (0:00:00.065) 0:01:44.505 ******* 2020-11-09T18:04:52.812 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:52 +0000 (0:00:00.033) 0:01:44.538 ******* 2020-11-09T18:04:52.840 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-11-09T18:04:52.841 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:52.894 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:52 +0000 (0:00:00.081) 0:01:44.619 ******* 2020-11-09T18:04:52.924 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-11-09T18:04:52.926 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:52.960 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:52 +0000 (0:00:00.066) 0:01:44.686 ******* 2020-11-09T18:04:52.988 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-11-09T18:04:52.990 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:53.024 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:53 +0000 (0:00:00.063) 0:01:44.749 ******* 2020-11-09T18:04:53.051 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-11-09T18:04:53.052 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:53.085 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:53 +0000 (0:00:00.061) 0:01:44.811 ******* 2020-11-09T18:04:53.092 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-11-09T18:04:53.094 INFO:teuthology.task.ansible.out:changed: [smithi051.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Mon 2020-11-09 18:01:50 UTC", "ActiveEnterTimestampMonotonic": "17816423", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-tmpfiles-setup.service remote-fs.target sysinit.target nss-lookup.target basic.target -.mount systemd-journald.socket network.target system.slice", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2020-11-09 18:01:50 UTC", "AssertTimestampMonotonic": "17332614", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2020-11-09 18:01:50 UTC", "ConditionTimestampMonotonic": "17332614", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2020-11-09T18:04:53.096 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1237", "ExecMainStartTimestamp": "Mon 2020-11-09 18:01:50 UTC", "ExecMainStartTimestampMonotonic": "17816411", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2020-11-09 18:01:50 UTC", "InactiveExitTimestampMonotonic": "17333783", "InvocationID": "0eb451b7125d4d5ab8e4efa2f7557eae", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2020-11-09T18:04:53.097 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128055", "LimitNPROCSoft": "128055", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128055", "LimitSIGPENDINGSoft": "128055", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1237", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit.target", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2020-11-09T18:04:53.099 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2020-11-09 18:01:50 UTC", "StateChangeTimestampMonotonic": "17816423", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2020-11-09 18:01:50 UTC", "WatchdogTimestampMonotonic": "17816422", "WatchdogUSec": "0"}} 2020-11-09T18:04:53.129 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:53 +0000 (0:00:00.043) 0:01:44.855 ******* 2020-11-09T18:04:53.136 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-11-09T18:04:53.138 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:53.178 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:53 +0000 (0:00:00.048) 0:01:44.903 ******* 2020-11-09T18:04:53.188 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-11-09T18:04:53.189 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:53.222 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:53 +0000 (0:00:00.044) 0:01:44.948 ******* 2020-11-09T18:04:53.231 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:53.264 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:53 +0000 (0:00:00.042) 0:01:44.990 ******* 2020-11-09T18:04:53.271 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-11-09T18:04:53.273 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:53.310 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:53 +0000 (0:00:00.045) 0:01:45.036 ******* 2020-11-09T18:04:53.318 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-11-09T18:04:53.320 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:53.353 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:53 +0000 (0:00:00.042) 0:01:45.078 ******* 2020-11-09T18:04:53.361 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-11-09T18:04:53.362 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:53.396 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:53 +0000 (0:00:00.043) 0:01:45.121 ******* 2020-11-09T18:04:53.403 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-11-09T18:04:53.405 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:53.441 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:53 +0000 (0:00:00.045) 0:01:45.167 ******* 2020-11-09T18:04:53.448 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-11-09T18:04:53.450 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:53.486 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:53 +0000 (0:00:00.045) 0:01:45.212 ******* 2020-11-09T18:04:53.495 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:53.496 INFO:teuthology.task.ansible.out: 2020-11-09T18:04:53.530 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:53 +0000 (0:00:00.044) 0:01:45.256 ******* 2020-11-09T18:04:53.537 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-11-09T18:04:53.539 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:53.572 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:53 +0000 (0:00:00.041) 0:01:45.298 ******* 2020-11-09T18:04:53.579 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-11-09T18:04:53.581 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:53.614 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:53 +0000 (0:00:00.041) 0:01:45.340 ******* 2020-11-09T18:04:53.620 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-11-09T18:04:53.622 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:53.656 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:53 +0000 (0:00:00.041) 0:01:45.382 ******* 2020-11-09T18:04:53.664 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-11-09T18:04:53.666 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:53.702 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:53 +0000 (0:00:00.046) 0:01:45.428 ******* 2020-11-09T18:04:53.709 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-11-09T18:04:53.711 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:53.745 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:53 +0000 (0:00:00.042) 0:01:45.470 ******* 2020-11-09T18:04:53.752 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-11-09T18:04:53.754 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:53.788 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:53 +0000 (0:00:00.042) 0:01:45.513 ******* 2020-11-09T18:04:53.794 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-11-09T18:04:53.796 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:53.830 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:53 +0000 (0:00:00.042) 0:01:45.556 ******* 2020-11-09T18:04:53.837 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-11-09T18:04:53.839 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:53.877 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:53 +0000 (0:00:00.046) 0:01:45.602 ******* 2020-11-09T18:04:53.885 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-09T18:04:53.886 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:53.922 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:53 +0000 (0:00:00.045) 0:01:45.648 ******* 2020-11-09T18:04:53.939 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-11-09T18:04:53.941 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:53.976 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:53 +0000 (0:00:00.053) 0:01:45.702 ******* 2020-11-09T18:04:53.983 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-11-09T18:04:53.985 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:54.021 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:54 +0000 (0:00:00.044) 0:01:45.746 ******* 2020-11-09T18:04:54.028 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-11-09T18:04:54.030 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:54.073 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:54 +0000 (0:00:00.052) 0:01:45.799 ******* 2020-11-09T18:04:54.081 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-11-09T18:04:54.085 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:54.117 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:54 +0000 (0:00:00.044) 0:01:45.843 ******* 2020-11-09T18:04:54.124 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2020-11-09T18:04:54.126 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:54.161 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:54 +0000 (0:00:00.043) 0:01:45.886 ******* 2020-11-09T18:04:54.167 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-11-09T18:04:54.168 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:54.204 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:54 +0000 (0:00:00.043) 0:01:45.930 ******* 2020-11-09T18:04:54.234 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:54.272 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:54 +0000 (0:00:00.068) 0:01:45.998 ******* 2020-11-09T18:04:54.303 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2020-11-09T18:04:54.305 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:54.358 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:54 +0000 (0:00:00.085) 0:01:46.083 ******* 2020-11-09T18:04:54.407 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:54 +0000 (0:00:00.049) 0:01:46.133 ******* 2020-11-09T18:04:54.417 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-11-09T18:04:54.419 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:54.469 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:54 +0000 (0:00:00.061) 0:01:46.194 ******* 2020-11-09T18:04:54.499 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:54.553 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:54 +0000 (0:00:00.084) 0:01:46.279 ******* 2020-11-09T18:04:54.581 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-11-09T18:04:54.584 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:54.615 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:54 +0000 (0:00:00.062) 0:01:46.341 ******* 2020-11-09T18:04:54.643 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-11-09T18:04:54.645 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:54.679 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:54 +0000 (0:00:00.063) 0:01:46.404 ******* 2020-11-09T18:04:54.708 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:54.756 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:54 +0000 (0:00:00.076) 0:01:46.481 ******* 2020-11-09T18:04:54.784 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-11-09T18:04:54.786 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:54.820 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:54 +0000 (0:00:00.063) 0:01:46.545 ******* 2020-11-09T18:04:54.852 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:54 +0000 (0:00:00.032) 0:01:46.578 ******* 2020-11-09T18:04:54.860 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-11-09T18:04:54.862 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:54.903 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:54 +0000 (0:00:00.050) 0:01:46.628 ******* 2020-11-09T18:04:54.911 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-11-09T18:04:54.913 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:54.950 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:54 +0000 (0:00:00.046) 0:01:46.675 ******* 2020-11-09T18:04:54.956 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-11-09T18:04:54.958 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:54.995 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:54 +0000 (0:00:00.045) 0:01:46.721 ******* 2020-11-09T18:04:55.026 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-11-09T18:04:55.027 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:55.075 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:55 +0000 (0:00:00.079) 0:01:46.801 ******* 2020-11-09T18:04:55.102 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:55.138 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:55 +0000 (0:00:00.061) 0:01:46.862 ******* 2020-11-09T18:04:55.164 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-11-09T18:04:55.166 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:55.228 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:55 +0000 (0:00:00.092) 0:01:46.954 ******* 2020-11-09T18:04:55.257 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:55.291 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:55 +0000 (0:00:00.063) 0:01:47.017 ******* 2020-11-09T18:04:55.309 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2020-11-09T18:04:55.311 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2020-11-09T18:04:55.344 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:55 +0000 (0:00:00.052) 0:01:47.070 ******* 2020-11-09T18:04:55.426 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2020-11-09T18:04:55.428 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2020-11-09T18:04:55.467 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:55 +0000 (0:00:00.122) 0:01:47.193 ******* 2020-11-09T18:04:55.712 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2020-11-09T18:04:55.758 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:55 +0000 (0:00:00.288) 0:01:47.481 ******* 2020-11-09T18:04:55.804 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-11-09T18:04:55.806 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2020-11-09T18:04:55.844 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:55 +0000 (0:00:00.088) 0:01:47.570 ******* 2020-11-09T18:04:55.876 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:55.921 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:55 +0000 (0:00:00.076) 0:01:47.647 ******* 2020-11-09T18:04:56.117 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2020-11-09T18:04:56.119 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2020-11-09T18:04:56.154 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:56 +0000 (0:00:00.233) 0:01:47.880 ******* 2020-11-09T18:04:56.258 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** ok: [smithi051.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2020-11-09T18:04:56.305 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:56 +0000 (0:00:00.150) 0:01:48.030 ******* 2020-11-09T18:04:56.315 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-11-09T18:04:56.316 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-09T18:04:56.352 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:56 +0000 (0:00:00.047) 0:01:48.078 ******* 2020-11-09T18:04:56.381 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-11-09T18:04:56.383 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:56.417 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:56 +0000 (0:00:00.065) 0:01:48.143 ******* 2020-11-09T18:04:56.445 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2020-11-09T18:04:56.447 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:56.483 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:56 +0000 (0:00:00.065) 0:01:48.209 ******* 2020-11-09T18:04:56.510 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-11-09T18:04:56.512 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:56.545 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:56 +0000 (0:00:00.061) 0:01:48.270 ******* 2020-11-09T18:04:56.579 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:56.611 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:56 +0000 (0:00:00.065) 0:01:48.336 ******* 2020-11-09T18:04:56.642 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2020-11-09T18:04:56.643 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:56.675 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:56 +0000 (0:00:00.064) 0:01:48.401 ******* 2020-11-09T18:04:56.682 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2020-11-09T18:04:56.684 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2020-11-09T18:04:56.720 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:56 +0000 (0:00:00.043) 0:01:48.445 ******* 2020-11-09T18:04:56.726 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-09T18:04:56.728 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-11-09T18:04:56.764 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:56 +0000 (0:00:00.044) 0:01:48.490 ******* 2020-11-09T18:04:56.816 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-11-09T18:04:56.818 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2020-11-09T18:04:56.827 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-11-09T18:04:56.839 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2020-11-09T18:04:56.840 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2020-11-09T18:04:56.850 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-11-09T18:04:56.861 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-11-09T18:04:56.872 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-11-09T18:04:56.874 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2020-11-09T18:04:56.884 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2020-11-09T18:04:56.895 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-11-09T18:04:56.897 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2020-11-09T18:04:56.908 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} skipping: [smithi078.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-11-09T18:04:56.918 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2020-11-09T18:04:56.921 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-11-09T18:04:56.968 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:56 +0000 (0:00:00.202) 0:01:48.692 ******* 2020-11-09T18:04:57.045 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2020-11-09T18:04:57.047 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2020-11-09T18:04:57.082 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:57 +0000 (0:00:00.115) 0:01:48.808 ******* 2020-11-09T18:04:57.168 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-11-09T18:04:57.169 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003837", "end": "2020-11-09 18:04:57.141210", "rc": 0, "start": "2020-11-09 18:04:57.137373", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-11-09T18:04:57.207 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:57 +0000 (0:00:00.125) 0:01:48.933 ******* 2020-11-09T18:04:57.226 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-11-09T18:04:57.228 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-09T18:04:57.263 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:57 +0000 (0:00:00.054) 0:01:48.988 ******* 2020-11-09T18:04:57.270 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2020-11-09T18:04:57.272 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:57.310 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:57 +0000 (0:00:00.047) 0:01:49.036 ******* 2020-11-09T18:04:57.318 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:57.358 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:57 +0000 (0:00:00.048) 0:01:49.084 ******* 2020-11-09T18:04:57.367 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-11-09T18:04:57.368 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:57.404 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:57 +0000 (0:00:00.045) 0:01:49.130 ******* 2020-11-09T18:04:57.414 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:57.447 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:57 +0000 (0:00:00.042) 0:01:49.173 ******* 2020-11-09T18:04:57.454 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-11-09T18:04:57.456 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:57.489 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:57 +0000 (0:00:00.042) 0:01:49.215 ******* 2020-11-09T18:04:57.496 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-11-09T18:04:57.497 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:57.531 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:57 +0000 (0:00:00.041) 0:01:49.257 ******* 2020-11-09T18:04:57.558 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-11-09T18:04:57.560 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:57.592 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:57 +0000 (0:00:00.060) 0:01:49.318 ******* 2020-11-09T18:04:57.629 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:57.669 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:57 +0000 (0:00:00.077) 0:01:49.395 ******* 2020-11-09T18:04:57.709 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-09T18:04:57.710 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-11-09T18:04:57.743 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:57 +0000 (0:00:00.074) 0:01:49.469 ******* 2020-11-09T18:04:57.781 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-11-09T18:04:57.783 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2020-11-09T18:04:57.793 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-11-09T18:04:57.804 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2020-11-09T18:04:57.805 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2020-11-09T18:04:57.815 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-11-09T18:04:57.816 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-11-09T18:04:57.826 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-11-09T18:04:57.837 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2020-11-09T18:04:57.839 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2020-11-09T18:04:57.848 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-11-09T18:04:57.861 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} skipping: [smithi051.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2020-11-09T18:04:57.863 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-11-09T18:04:57.873 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2020-11-09T18:04:57.886 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-11-09T18:04:57.923 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:57 +0000 (0:00:00.179) 0:01:49.648 ******* 2020-11-09T18:04:58.119 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** changed: [smithi051.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003602", "end": "2020-11-09 18:04:58.091392", "rc": 0, "start": "2020-11-09 18:04:58.087790", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-11-09T18:04:58.160 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:58 +0000 (0:00:00.236) 0:01:49.885 ******* 2020-11-09T18:04:58.188 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2020-11-09T18:04:58.189 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:58.228 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:58 +0000 (0:00:00.068) 0:01:49.954 ******* 2020-11-09T18:04:58.256 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-11-09T18:04:58.258 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:58.291 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:58 +0000 (0:00:00.063) 0:01:50.017 ******* 2020-11-09T18:04:58.318 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-11-09T18:04:58.320 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:58.352 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:58 +0000 (0:00:00.060) 0:01:50.078 ******* 2020-11-09T18:04:58.379 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2020-11-09T18:04:58.380 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"cache_update_time": 1604944981, "cache_updated": false, "changed": false} 2020-11-09T18:04:58.414 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:58 +0000 (0:00:00.062) 0:01:50.140 ******* 2020-11-09T18:04:58.442 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2020-11-09T18:04:58.443 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:58.478 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:58 +0000 (0:00:00.063) 0:01:50.203 ******* 2020-11-09T18:04:58.480 INFO:teuthology.task.ansible.out: 2020-11-09T18:04:58.518 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ ok: [smithi078.front.sepia.ceph.com] => (item={'block_used': 13640638, 'uuid': '1e9a17af-0168-46be-b05d-6a296317ab15', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226454470, 'size_available': 927557509120, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113945, 'block_size': 4096, 'inode_available': 60941031}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226454470, "block_size": 4096, "block_total": 240095108, "block_used": 13640638, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941031, "inode_total": 61054976, "inode_used": 113945, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927557509120, "size_total": 983429562368, "uuid": "1e9a17af-0168-46be-b05d-6a296317ab15"}} 2020-11-09T18:04:58.555 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:58 +0000 (0:00:00.077) 0:01:50.281 ******* 2020-11-09T18:04:58.699 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** changed: [smithi078.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.009777", "end": "2020-11-09 18:04:59.092783", "rc": 0, "start": "2020-11-09 18:04:59.083006", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-09T18:04:58.737 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:58 +0000 (0:00:00.182) 0:01:50.463 ******* 2020-11-09T18:04:58.775 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-11-09T18:04:58.777 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'block_used': 13640638, 'uuid': '1e9a17af-0168-46be-b05d-6a296317ab15', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226454470, 'size_available': 927557509120, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113945, 'block_size': 4096, 'inode_available': 60941031}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226454470, "block_size": 4096, "block_total": 240095108, "block_used": 13640638, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941031, "inode_total": 61054976, "inode_used": 113945, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927557509120, "size_total": 983429562368, "uuid": "1e9a17af-0168-46be-b05d-6a296317ab15"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:58.814 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:58 +0000 (0:00:00.076) 0:01:50.540 ******* 2020-11-09T18:04:58.841 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-11-09T18:04:58.878 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:58 +0000 (0:00:00.064) 0:01:50.604 ******* 2020-11-09T18:04:58.916 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-11-09T18:04:58.918 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:59.267 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* ok: [smithi051.front.sepia.ceph.com] => {"cache_update_time": 1604944975, "cache_updated": false, "changed": false} 2020-11-09T18:04:59.302 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:59 +0000 (0:00:00.423) 0:01:51.028 ******* 2020-11-09T18:04:59.329 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:59.366 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:59 +0000 (0:00:00.063) 0:01:51.091 ******* 2020-11-09T18:04:59.407 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-11-09T18:04:59.408 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item={'block_used': 13640652, 'uuid': '1e9a17af-0168-46be-b05d-6a296317ab15', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226454456, 'size_available': 927557451776, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113946, 'block_size': 4096, 'inode_available': 60941030}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226454456, "block_size": 4096, "block_total": 240095108, "block_used": 13640652, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941030, "inode_total": 61054976, "inode_used": 113946, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927557451776, "size_total": 983429562368, "uuid": "1e9a17af-0168-46be-b05d-6a296317ab15"}} 2020-11-09T18:04:59.443 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:59 +0000 (0:00:00.077) 0:01:51.169 ******* 2020-11-09T18:04:59.576 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-11-09T18:04:59.578 INFO:teuthology.task.ansible.out:changed: [smithi051.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.009636", "end": "2020-11-09 18:04:59.226241", "rc": 0, "start": "2020-11-09 18:04:59.216605", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-09T18:04:59.611 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:59 +0000 (0:00:00.167) 0:01:51.336 ******* 2020-11-09T18:04:59.649 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-11-09T18:04:59.650 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'block_used': 13640652, 'uuid': '1e9a17af-0168-46be-b05d-6a296317ab15', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226454456, 'size_available': 927557451776, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113946, 'block_size': 4096, 'inode_available': 60941030}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226454456, "block_size": 4096, "block_total": 240095108, "block_used": 13640652, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941030, "inode_total": 61054976, "inode_used": 113946, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927557451776, "size_total": 983429562368, "uuid": "1e9a17af-0168-46be-b05d-6a296317ab15"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:04:59.686 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:59 +0000 (0:00:00.075) 0:01:51.411 ******* 2020-11-09T18:04:59.713 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-11-09T18:04:59.748 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:04:59 +0000 (0:00:00.062) 0:01:51.474 ******* 2020-11-09T18:04:59.809 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-11-09T18:04:59.810 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:01.065 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-v03rI10oYDwy3Rz5nBKOnEpYyQruUfhoS7n21K2iY07qBhLiLXizli4QBaJclpff'], 'uuids': ['1b3e3df4-0e3f-4b6e-87e1-7c3c31f62879']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:02.007621", "end": "2020-11-09 18:05:01.439359", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-v03rI10oYDwy3Rz5nBKOnEpYyQruUfhoS7n21K2iY07qBhLiLXizli4QBaJclpff"], "labels": [], "masters": [], "uuids": ["1b3e3df4-0e3f-4b6e-87e1-7c3c31f62879"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 18:04:59.431738", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-09T18:05:01.077 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:01.078 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:01.942 INFO:teuthology.task.ansible.out:changed: [smithi051.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-nnodJAeBrmQATKwnQMfQhlMCFNAred4U2dWp37I40DLh5sIvS0d1pTRCxSTA4pVl'], 'uuids': ['be825212-dded-4c78-8317-e00a27aa38de']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:02.015484", "end": "2020-11-09 18:05:01.579908", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-nnodJAeBrmQATKwnQMfQhlMCFNAred4U2dWp37I40DLh5sIvS0d1pTRCxSTA4pVl"], "labels": [], "masters": [], "uuids": ["be825212-dded-4c78-8317-e00a27aa38de"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 18:04:59.564424", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-09T18:05:01.954 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:01.966 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:02.333 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-v03rI10oYDwy3Rz5nBKOnEpYyQruUfhoUVNALBdW8krN2Bf5qc8h2aVrN7EjFScW'], 'uuids': ['0152ec84-dcff-48d5-ba5c-17677ed8d8ae']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.127836", "end": "2020-11-09 18:05:02.715210", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-v03rI10oYDwy3Rz5nBKOnEpYyQruUfhoUVNALBdW8krN2Bf5qc8h2aVrN7EjFScW"], "labels": [], "masters": [], "uuids": ["0152ec84-dcff-48d5-ba5c-17677ed8d8ae"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 18:05:01.587374", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-09T18:05:02.345 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LGR8', 'wwn-0x5000c500918e79d4'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '1e9a17af-0168-46be-b05d-6a296317ab15', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LGR8-part1', 'wwn-0x5000c500918e79d4-part1'], 'uuids': ['1e9a17af-0168-46be-b05d-6a296317ab15']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500918e79d4', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5LGR8', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LGR8", "wwn-0x5000c500918e79d4"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LGR8-part1", "wwn-0x5000c500918e79d4-part1"], "labels": [], "masters": [], "uuids": ["1e9a17af-0168-46be-b05d-6a296317ab15"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "1e9a17af-0168-46be-b05d-6a296317ab15"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5LGR8", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500918e79d4"}}, "skip_reason": 2020-11-09T18:05:02.347 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-11-09T18:05:03.241 INFO:teuthology.task.ansible.out:changed: [smithi051.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-nnodJAeBrmQATKwnQMfQhlMCFNAred4U5QCbKLc4JfGrDg9DsovE24jfggtaMCvc'], 'uuids': ['a25bb6e3-f7e0-413a-bdda-cc70e7db3fd1']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.148936", "end": "2020-11-09 18:05:02.878452", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-nnodJAeBrmQATKwnQMfQhlMCFNAred4U5QCbKLc4JfGrDg9DsovE24jfggtaMCvc"], "labels": [], "masters": [], "uuids": ["a25bb6e3-f7e0-413a-bdda-cc70e7db3fd1"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 18:05:01.729516", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-09T18:05:03.252 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HPJX', 'wwn-0x5000c500871aa859'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '1e9a17af-0168-46be-b05d-6a296317ab15', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HPJX-part1', 'wwn-0x5000c500871aa859-part1'], 'uuids': ['1e9a17af-0168-46be-b05d-6a296317ab15']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500871aa859', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4HPJX', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HPJX", "wwn-0x5000c500871aa859"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HPJX-part1", "wwn-0x5000c500871aa859-part1"], "labels": [], "masters": [], "uuids": ["1e9a17af-0168-46be-b05d-6a296317ab15"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "1e9a17af-0168-46be-b05d-6a296317ab15"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HPJX", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871aa859"}}, "skip_reason": 2020-11-09T18:05:03.254 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-11-09T18:05:03.655 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-N4gBf9-oKOE-XUKW-V5Rx-U5Co-2NVa-mKrgEM', 'nvme-INTEL_SSDPEDMD400G4_PHFT620400QR400BGN', 'nvme-nvme.8086-50484654363230343030515234303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.186180", "end": "2020-11-09 18:05:04.032843", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-N4gBf9-oKOE-XUKW-V5Rx-U5Co-2NVa-mKrgEM", "nvme-INTEL_SSDPEDMD400G4_PHFT620400QR400BGN", "nvme-nvme.8086-50484654363230343030515234303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 18:05:02.846663", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2020-11-09T18:05:03.657 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-09T18:05:03.666 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:03.678 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:04.554 INFO:teuthology.task.ansible.out:changed: [smithi051.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '3125627568', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-MxYA1C-P7cB-r2Lm-3hjT-q6ld-dfp5-WSPZy3', 'nvme-INTEL_SSDPEDMD016T4_CVFT6011006A1P6DGN', 'nvme-nvme.8086-43564654363031313030364131503644474e-494e54454c205353445045444d443031365434-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD016T4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '1.46 TB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.176535", "end": "2020-11-09 18:05:04.197128", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-MxYA1C-P7cB-r2Lm-3hjT-q6ld-dfp5-WSPZy3", "nvme-INTEL_SSDPEDMD016T4_CVFT6011006A1P6DGN", "nvme-nvme.8086-43564654363031313030364131503644474e-494e54454c205353445045444d443031365434-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD016T4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "3125627568", "sectorsize": "512", "size": "1.46 TB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 18:05:03.020593", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! Yo 2020-11-09T18:05:04.556 INFO:teuthology.task.ansible.out:u may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-09T18:05:04.566 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:04.578 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:05.069 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-v03rI10oYDwy3Rz5nBKOnEpYyQruUfhoKx4tN4H1LVtquK03va1TE6eQKVODQOOO'], 'uuids': ['483c391a-7121-4c07-9f9a-17361be5af31']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.270356", "end": "2020-11-09 18:05:05.443879", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-v03rI10oYDwy3Rz5nBKOnEpYyQruUfhoKx4tN4H1LVtquK03va1TE6eQKVODQOOO"], "labels": [], "masters": [], "uuids": ["483c391a-7121-4c07-9f9a-17361be5af31"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 18:05:04.173523", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-09T18:05:05.070 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:05.092 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:05.835 INFO:teuthology.task.ansible.out:changed: [smithi051.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-nnodJAeBrmQATKwnQMfQhlMCFNAred4U7XhuaZKTjBQ35srCtarwYzltIIveVgdh'], 'uuids': ['451dbd20-ab54-44ba-9415-cfa6a56a2eb8']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.146105", "end": "2020-11-09 18:05:05.474474", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-nnodJAeBrmQATKwnQMfQhlMCFNAred4U7XhuaZKTjBQ35srCtarwYzltIIveVgdh"], "labels": [], "masters": [], "uuids": ["451dbd20-ab54-44ba-9415-cfa6a56a2eb8"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 18:05:04.328369", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-09T18:05:05.847 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:05.858 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:06.558 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-v03rI10oYDwy3Rz5nBKOnEpYyQruUfhoo9PoKcEKqvG0p9R1dUTuvhthqkz4wYiz'], 'uuids': ['764f2c96-7ac7-408a-81fd-6838c9417f33']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.339542", "end": "2020-11-09 18:05:06.930301", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-v03rI10oYDwy3Rz5nBKOnEpYyQruUfhoo9PoKcEKqvG0p9R1dUTuvhthqkz4wYiz"], "labels": [], "masters": [], "uuids": ["764f2c96-7ac7-408a-81fd-6838c9417f33"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 18:05:05.590759", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-09T18:05:06.570 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:07.094 INFO:teuthology.task.ansible.out:changed: [smithi051.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-nnodJAeBrmQATKwnQMfQhlMCFNAred4UduPrwJpcV5vUw6zeyIVKPKPucwTOSvN8'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.117370", "end": "2020-11-09 18:05:06.731542", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-nnodJAeBrmQATKwnQMfQhlMCFNAred4UduPrwJpcV5vUw6zeyIVKPKPucwTOSvN8"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 18:05:05.614172", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-09T18:05:07.105 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:07.989 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-v03rI10oYDwy3Rz5nBKOnEpYyQruUfhoF5jMafQxgNocf32Rccg2q3Cv7A1DKw9r'], 'uuids': ['ccc2f903-7c6d-48c7-8719-24315c5acd6d']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.303308", "end": "2020-11-09 18:05:08.372569", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-v03rI10oYDwy3Rz5nBKOnEpYyQruUfhoF5jMafQxgNocf32Rccg2q3Cv7A1DKw9r"], "labels": [], "masters": [], "uuids": ["ccc2f903-7c6d-48c7-8719-24315c5acd6d"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 18:05:07.069261", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-09T18:05:08.033 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:08 +0000 (0:00:08.284) 0:01:59.759 ******* 2020-11-09T18:05:08.083 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-11-09T18:05:08.085 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:08.340 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** changed: [smithi051.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-nnodJAeBrmQATKwnQMfQhlMCFNAred4UxKg3H66Z2KTwjfwQeRtBkprKwfw49Mxd'], 'uuids': ['81f260b0-0f07-4981-9eef-0423cd49a2ae']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.112450", "end": "2020-11-09 18:05:07.980455", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-nnodJAeBrmQATKwnQMfQhlMCFNAred4UxKg3H66Z2KTwjfwQeRtBkprKwfw49Mxd"], "labels": [], "masters": [], "uuids": ["81f260b0-0f07-4981-9eef-0423cd49a2ae"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 18:05:06.868005", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk u 2020-11-09T18:05:08.341 INFO:teuthology.task.ansible.out:sing fdisk or", "other utilities."]} 2020-11-09T18:05:08.387 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:08 +0000 (0:00:00.354) 0:02:00.113 ******* 2020-11-09T18:05:08.397 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-11-09T18:05:08.399 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-v03rI10oYDwy3Rz5nBKOnEpYyQruUfhoS7n21K2iY07qBhLiLXizli4QBaJclpff'], 'uuids': ['1b3e3df4-0e3f-4b6e-87e1-7c3c31f62879']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.165207", "end": "2020-11-09 18:05:08.754241", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-v03rI10oYDwy3Rz5nBKOnEpYyQruUfhoS7n21K2iY07qBhLiLXizli4QBaJclpff"], "labels": [], "masters": [], "uuids": ["1b3e3df4-0e3f-4b6e-87e1-7c3c31f62879"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 18:05:08.589034", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.162396 s, 710 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.162396 s, 710 MB/s"], "stdout": "", "stdout_lines": []} skipping: [smithi078.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': 2020-11-09T18:05:08.402 INFO:teuthology.task.ansible.out: '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} skipping: [smithi078.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:08.433 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:08.671 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-v03rI10oYDwy3Rz5nBKOnEpYyQruUfhoUVNALBdW8krN2Bf5qc8h2aVrN7EjFScW'], 'uuids': ['0152ec84-dcff-48d5-ba5c-17677ed8d8ae']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.150664", "end": "2020-11-09 18:05:09.049714", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-v03rI10oYDwy3Rz5nBKOnEpYyQruUfhoUVNALBdW8krN2Bf5qc8h2aVrN7EjFScW"], "labels": [], "masters": [], "uuids": ["0152ec84-dcff-48d5-ba5c-17677ed8d8ae"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 18:05:08.899050", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.147939 s, 780 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.147939 s, 780 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-09T18:05:08.673 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LGR8', 'wwn-0x5000c500918e79d4'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '1e9a17af-0168-46be-b05d-6a296317ab15', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LGR8-part1', 'wwn-0x5000c500918e79d4-part1'], 'uuids': ['1e9a17af-0168-46be-b05d-6a296317ab15']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500918e79d4', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5LGR8', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LGR8", "wwn-0x5000c500918e79d4"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LGR8-part1", "wwn-0x5000c500918e79d4-part1"], "labels": [], "masters": [], "uuids": ["1e9a17af-0168-46be-b05d-6a296317ab15"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "1e9a17af-0168-46be-b05d-6a296317ab15"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5LGR8", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500918e79d4"}}, "skip_reason": 2020-11-09T18:05:08.675 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-11-09T18:05:08.696 INFO:teuthology.task.ansible.out:changed: [smithi051.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-nnodJAeBrmQATKwnQMfQhlMCFNAred4U2dWp37I40DLh5sIvS0d1pTRCxSTA4pVl'], 'uuids': ['be825212-dded-4c78-8317-e00a27aa38de']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.122343", "end": "2020-11-09 18:05:08.323936", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-nnodJAeBrmQATKwnQMfQhlMCFNAred4U2dWp37I40DLh5sIvS0d1pTRCxSTA4pVl"], "labels": [], "masters": [], "uuids": ["be825212-dded-4c78-8317-e00a27aa38de"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 18:05:08.201593", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.117553 s, 981 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.117553 s, 981 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-09T18:05:08.697 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:08.709 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:08.850 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-N4gBf9-oKOE-XUKW-V5Rx-U5Co-2NVa-mKrgEM', 'nvme-INTEL_SSDPEDMD400G4_PHFT620400QR400BGN', 'nvme-nvme.8086-50484654363230343030515234303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.047083", "end": "2020-11-09 18:05:09.221972", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-N4gBf9-oKOE-XUKW-V5Rx-U5Co-2NVa-mKrgEM", "nvme-INTEL_SSDPEDMD400G4_PHFT620400QR400BGN", "nvme-nvme.8086-50484654363230343030515234303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 18:05:09.174889", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0425408 s, 2.7 GB/s", "std 2020-11-09T18:05:08.852 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0425408 s, 2.7 GB/s"], "stdout": "", "stdout_lines": []} 2020-11-09T18:05:08.861 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:08.873 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:08.959 INFO:teuthology.task.ansible.out:changed: [smithi051.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-nnodJAeBrmQATKwnQMfQhlMCFNAred4U5QCbKLc4JfGrDg9DsovE24jfggtaMCvc'], 'uuids': ['a25bb6e3-f7e0-413a-bdda-cc70e7db3fd1']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.120476", "end": "2020-11-09 18:05:08.597155", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-nnodJAeBrmQATKwnQMfQhlMCFNAred4U5QCbKLc4JfGrDg9DsovE24jfggtaMCvc"], "labels": [], "masters": [], "uuids": ["a25bb6e3-f7e0-413a-bdda-cc70e7db3fd1"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 18:05:08.476679", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.115771 s, 996 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.115771 s, 996 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-09T18:05:08.970 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HPJX', 'wwn-0x5000c500871aa859'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '1e9a17af-0168-46be-b05d-6a296317ab15', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HPJX-part1', 'wwn-0x5000c500871aa859-part1'], 'uuids': ['1e9a17af-0168-46be-b05d-6a296317ab15']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500871aa859', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4HPJX', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HPJX", "wwn-0x5000c500871aa859"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HPJX-part1", "wwn-0x5000c500871aa859-part1"], "labels": [], "masters": [], "uuids": ["1e9a17af-0168-46be-b05d-6a296317ab15"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "1e9a17af-0168-46be-b05d-6a296317ab15"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HPJX", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871aa859"}}, "skip_reason": 2020-11-09T18:05:08.972 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-11-09T18:05:09.150 INFO:teuthology.task.ansible.out:changed: [smithi051.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '3125627568', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-MxYA1C-P7cB-r2Lm-3hjT-q6ld-dfp5-WSPZy3', 'nvme-INTEL_SSDPEDMD016T4_CVFT6011006A1P6DGN', 'nvme-nvme.8086-43564654363031313030364131503644474e-494e54454c205353445045444d443031365434-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD016T4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '1.46 TB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.051048", "end": "2020-11-09 18:05:08.788830", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-MxYA1C-P7cB-r2Lm-3hjT-q6ld-dfp5-WSPZy3", "nvme-INTEL_SSDPEDMD016T4_CVFT6011006A1P6DGN", "nvme-nvme.8086-43564654363031313030364131503644474e-494e54454c205353445045444d443031365434-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD016T4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "3125627568", "sectorsize": "512", "size": "1.46 TB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 18:05:08.737782", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0461891 s, 2.5 GB/s", "stder 2020-11-09T18:05:09.152 INFO:teuthology.task.ansible.out:r_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0461891 s, 2.5 GB/s"], "stdout": "", "stdout_lines": []} changed: [smithi078.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-v03rI10oYDwy3Rz5nBKOnEpYyQruUfhoKx4tN4H1LVtquK03va1TE6eQKVODQOOO'], 'uuids': ['483c391a-7121-4c07-9f9a-17361be5af31']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.147922", "end": "2020-11-09 18:05:09.526514", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-v03rI10oYDwy3Rz5nBKOnEpYyQruUfhoKx4tN4H1LVtquK03va1TE6eQKVODQOOO"], "labels": [], "masters": [], "uuids": ["483c391a-7121-4c07-9f9a-17361be5af31"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 18:05:09.378592", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.145114 s, 795 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.145114 s, 795 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-09T18:05:09.163 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:09.164 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:09.165 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:09.176 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:09.401 INFO:teuthology.task.ansible.out:changed: [smithi051.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-nnodJAeBrmQATKwnQMfQhlMCFNAred4U7XhuaZKTjBQ35srCtarwYzltIIveVgdh'], 'uuids': ['451dbd20-ab54-44ba-9415-cfa6a56a2eb8']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.121714", "end": "2020-11-09 18:05:09.044367", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-nnodJAeBrmQATKwnQMfQhlMCFNAred4U7XhuaZKTjBQ35srCtarwYzltIIveVgdh"], "labels": [], "masters": [], "uuids": ["451dbd20-ab54-44ba-9415-cfa6a56a2eb8"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 18:05:08.922653", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.117735 s, 980 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.117735 s, 980 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-09T18:05:09.412 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:09.424 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:09.458 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-v03rI10oYDwy3Rz5nBKOnEpYyQruUfhoo9PoKcEKqvG0p9R1dUTuvhthqkz4wYiz'], 'uuids': ['764f2c96-7ac7-408a-81fd-6838c9417f33']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.155856", "end": "2020-11-09 18:05:09.833293", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-v03rI10oYDwy3Rz5nBKOnEpYyQruUfhoo9PoKcEKqvG0p9R1dUTuvhthqkz4wYiz"], "labels": [], "masters": [], "uuids": ["764f2c96-7ac7-408a-81fd-6838c9417f33"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 18:05:09.677437", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.151316 s, 762 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.151316 s, 762 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-09T18:05:09.469 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:09.662 INFO:teuthology.task.ansible.out:changed: [smithi051.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-nnodJAeBrmQATKwnQMfQhlMCFNAred4UduPrwJpcV5vUw6zeyIVKPKPucwTOSvN8'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.122657", "end": "2020-11-09 18:05:09.305883", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-nnodJAeBrmQATKwnQMfQhlMCFNAred4UduPrwJpcV5vUw6zeyIVKPKPucwTOSvN8"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 18:05:09.183226", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.11777 s, 979 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.11777 s, 979 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-09T18:05:09.684 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:09.739 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-v03rI10oYDwy3Rz5nBKOnEpYyQruUfhoF5jMafQxgNocf32Rccg2q3Cv7A1DKw9r'], 'uuids': ['ccc2f903-7c6d-48c7-8719-24315c5acd6d']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.158850", "end": "2020-11-09 18:05:10.117539", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-v03rI10oYDwy3Rz5nBKOnEpYyQruUfhoF5jMafQxgNocf32Rccg2q3Cv7A1DKw9r"], "labels": [], "masters": [], "uuids": ["ccc2f903-7c6d-48c7-8719-24315c5acd6d"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 18:05:09.958689", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.155443 s, 742 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.155443 s, 742 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-09T18:05:09.792 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:09 +0000 (0:00:01.404) 0:02:01.518 ******* 2020-11-09T18:05:09.929 INFO:teuthology.task.ansible.out:changed: [smithi051.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-nnodJAeBrmQATKwnQMfQhlMCFNAred4UxKg3H66Z2KTwjfwQeRtBkprKwfw49Mxd'], 'uuids': ['81f260b0-0f07-4981-9eef-0423cd49a2ae']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.125737", "end": "2020-11-09 18:05:09.565345", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-nnodJAeBrmQATKwnQMfQhlMCFNAred4UxKg3H66Z2KTwjfwQeRtBkprKwfw49Mxd"], "labels": [], "masters": [], "uuids": ["81f260b0-0f07-4981-9eef-0423cd49a2ae"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 18:05:09.439608", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.121006 s, 953 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.121006 s, 953 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-09T18:05:09.974 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:09 +0000 (0:00:00.182) 0:02:01.700 ******* 2020-11-09T18:05:10.060 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** changed: [smithi078.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.120780", "end": "2020-11-09 18:05:10.440557", "rc": 0, "start": "2020-11-09 18:05:10.319777", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-09T18:05:10.098 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:10 +0000 (0:00:00.123) 0:02:01.824 ******* 2020-11-09T18:05:10.252 INFO:teuthology.task.ansible.out:changed: [smithi051.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.127313", "end": "2020-11-09 18:05:09.884976", "rc": 0, "start": "2020-11-09 18:05:09.757663", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-09T18:05:10.291 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:10 +0000 (0:00:00.192) 0:02:02.017 ******* 2020-11-09T18:05:10.343 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-11-09T18:05:10.345 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.116258", "end": "2020-11-09 18:05:10.721103", "rc": 0, "start": "2020-11-09 18:05:10.604845", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-09T18:05:10.383 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:10 +0000 (0:00:00.090) 0:02:02.107 ******* 2020-11-09T18:05:10.412 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-11-09T18:05:10.449 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:10 +0000 (0:00:00.066) 0:02:02.174 ******* 2020-11-09T18:05:10.475 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-11-09T18:05:10.477 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Check for physical volumes] *********************************** changed: [smithi051.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.049499", "end": "2020-11-09 18:05:10.119666", "rc": 0, "start": "2020-11-09 18:05:10.070167", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-09T18:05:10.511 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:10 +0000 (0:00:00.062) 0:02:02.237 ******* 2020-11-09T18:05:10.545 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:10 +0000 (0:00:00.034) 0:02:02.271 ******* 2020-11-09T18:05:10.551 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-11-09T18:05:10.589 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:10 +0000 (0:00:00.044) 0:02:02.315 ******* 2020-11-09T18:05:10.598 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:10.635 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:10 +0000 (0:00:00.045) 0:02:02.360 ******* 2020-11-09T18:05:10.641 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-11-09T18:05:10.643 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:10.681 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:10 +0000 (0:00:00.046) 0:02:02.407 ******* 2020-11-09T18:05:10.688 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-11-09T18:05:10.689 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:10.728 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:10 +0000 (0:00:00.046) 0:02:02.454 ******* 2020-11-09T18:05:10.734 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-11-09T18:05:10.736 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:10.771 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:10 +0000 (0:00:00.042) 0:02:02.497 ******* 2020-11-09T18:05:10.777 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-11-09T18:05:10.780 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:10.817 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:10 +0000 (0:00:00.046) 0:02:02.543 ******* 2020-11-09T18:05:10.825 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:10.870 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:10 +0000 (0:00:00.052) 0:02:02.595 ******* 2020-11-09T18:05:10.900 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-11-09T18:05:10.901 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:10.935 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:10 +0000 (0:00:00.065) 0:02:02.661 ******* 2020-11-09T18:05:10.953 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-11-09T18:05:10.955 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-09T18:05:10.991 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:10 +0000 (0:00:00.054) 0:02:02.716 ******* 2020-11-09T18:05:11.040 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ skipping: [smithi078.front.sepia.ceph.com] => (item={'block_used': 13640638, 'uuid': '1e9a17af-0168-46be-b05d-6a296317ab15', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226454470, 'size_available': 927557509120, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113945, 'block_size': 4096, 'inode_available': 60941031}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226454470, "block_size": 4096, "block_total": 240095108, "block_used": 13640638, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941031, "inode_total": 61054976, "inode_used": 113945, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927557509120, "size_total": 983429562368, "uuid": "1e9a17af-0168-46be-b05d-6a296317ab15"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:11.076 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:11 +0000 (0:00:00.085) 0:02:02.801 ******* 2020-11-09T18:05:11.077 INFO:teuthology.task.ansible.out: 2020-11-09T18:05:11.084 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ ok: [smithi051.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-09T18:05:11.122 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:11 +0000 (0:00:00.046) 0:02:02.848 ******* 2020-11-09T18:05:11.129 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2020-11-09T18:05:11.131 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:11.170 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:11 +0000 (0:00:00.047) 0:02:02.895 ******* 2020-11-09T18:05:11.177 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-11-09T18:05:11.179 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'block_used': 13640652, 'uuid': '1e9a17af-0168-46be-b05d-6a296317ab15', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226454456, 'size_available': 927557451776, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113946, 'block_size': 4096, 'inode_available': 60941030}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226454456, "block_size": 4096, "block_total": 240095108, "block_used": 13640652, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941030, "inode_total": 61054976, "inode_used": 113946, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927557451776, "size_total": 983429562368, "uuid": "1e9a17af-0168-46be-b05d-6a296317ab15"}, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:11.220 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:11 +0000 (0:00:00.049) 0:02:02.945 ******* 2020-11-09T18:05:11.227 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-09T18:05:11.228 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:11.265 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:11 +0000 (0:00:00.045) 0:02:02.990 ******* 2020-11-09T18:05:11.272 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2020-11-09T18:05:11.273 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:11.308 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:11 +0000 (0:00:00.043) 0:02:03.034 ******* 2020-11-09T18:05:11.315 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-09T18:05:11.317 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:11.356 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:11 +0000 (0:00:00.048) 0:02:03.082 ******* 2020-11-09T18:05:11.363 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-09T18:05:11.365 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:11.399 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:11 +0000 (0:00:00.042) 0:02:03.125 ******* 2020-11-09T18:05:11.406 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-11-09T18:05:11.408 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:11.446 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:11 +0000 (0:00:00.046) 0:02:03.172 ******* 2020-11-09T18:05:11.452 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-09T18:05:11.454 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:11.488 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:11 +0000 (0:00:00.042) 0:02:03.214 ******* 2020-11-09T18:05:11.495 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:11.530 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:11 +0000 (0:00:00.041) 0:02:03.256 ******* 2020-11-09T18:05:11.544 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:11.580 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:11 +0000 (0:00:00.049) 0:02:03.305 ******* 2020-11-09T18:05:11.587 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2020-11-09T18:05:11.589 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:11.629 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:11 +0000 (0:00:00.048) 0:02:03.354 ******* 2020-11-09T18:05:11.634 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2020-11-09T18:05:11.636 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:11.671 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:11 +0000 (0:00:00.043) 0:02:03.397 ******* 2020-11-09T18:05:11.699 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2020-11-09T18:05:11.701 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:11.735 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:11 +0000 (0:00:00.063) 0:02:03.461 ******* 2020-11-09T18:05:12.220 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2020-11-09T18:05:12.221 INFO:teuthology.task.ansible.out:changed: [smithi051.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-11-09T18:05:12.256 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:12 +0000 (0:00:00.520) 0:02:03.981 ******* 2020-11-09T18:05:12.265 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-11-09T18:05:12.302 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:12 +0000 (0:00:00.046) 0:02:04.027 ******* 2020-11-09T18:05:12.809 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-11-09T18:05:12.811 INFO:teuthology.task.ansible.out:changed: [smithi051.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-09T18:05:12.832 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-09T18:05:13.078 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} changed: [smithi051.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-09T18:05:13.293 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-09T18:05:13.326 INFO:teuthology.task.ansible.out:changed: [smithi051.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-09T18:05:13.514 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-09T18:05:13.582 INFO:teuthology.task.ansible.out:changed: [smithi051.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-09T18:05:13.745 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-09T18:05:13.784 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:13 +0000 (0:00:01.482) 0:02:05.510 ******* 2020-11-09T18:05:13.840 INFO:teuthology.task.ansible.out:changed: [smithi051.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-09T18:05:13.880 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:13 +0000 (0:00:00.095) 0:02:05.606 ******* 2020-11-09T18:05:13.930 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-11-09T18:05:13.932 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-11-09T18:05:13.980 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:13 +0000 (0:00:00.099) 0:02:05.705 ******* 2020-11-09T18:05:14.011 INFO:teuthology.task.ansible.out:changed: [smithi051.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-11-09T18:05:14.050 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:14 +0000 (0:00:00.070) 0:02:05.776 ******* 2020-11-09T18:05:14.135 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-11-09T18:05:14.136 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-09T18:05:14.211 INFO:teuthology.task.ansible.out:changed: [smithi051.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-09T18:05:14.254 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-09T18:05:14.330 INFO:teuthology.task.ansible.out:changed: [smithi051.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-09T18:05:14.374 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-09T18:05:14.450 INFO:teuthology.task.ansible.out:changed: [smithi051.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-09T18:05:14.493 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-09T18:05:14.505 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:14.543 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:14 +0000 (0:00:00.493) 0:02:06.269 ******* 2020-11-09T18:05:14.562 INFO:teuthology.task.ansible.out:changed: [smithi051.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-09T18:05:14.573 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:14.575 INFO:teuthology.task.ansible.out: 2020-11-09T18:05:14.614 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:14 +0000 (0:00:00.070) 0:02:06.339 ******* 2020-11-09T18:05:14.689 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2020-11-09T18:05:14.725 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:14 +0000 (0:00:00.111) 0:02:06.451 ******* 2020-11-09T18:05:14.755 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} TASK [testnode : Set xfs_opts on newer OSes] *********************************** ok: [smithi078.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2020-11-09T18:05:14.792 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:14 +0000 (0:00:00.066) 0:02:06.517 ******* 2020-11-09T18:05:14.831 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:14 +0000 (0:00:00.038) 0:02:06.556 ******* 2020-11-09T18:05:14.838 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2020-11-09T18:05:14.875 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:14 +0000 (0:00:00.043) 0:02:06.600 ******* 2020-11-09T18:05:15.316 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* changed: [smithi078.front.sepia.ceph.com] => {"changed": true} changed: [smithi051.front.sepia.ceph.com] => {"changed": true} 2020-11-09T18:05:15.353 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:15 +0000 (0:00:00.479) 0:02:07.079 ******* 2020-11-09T18:05:15.415 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:15 +0000 (0:00:00.061) 0:02:07.140 ******* 2020-11-09T18:05:15.779 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* changed: [smithi078.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-11-09T18:05:15.815 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:15 +0000 (0:00:00.400) 0:02:07.541 ******* 2020-11-09T18:05:15.822 INFO:teuthology.task.ansible.out:changed: [smithi051.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-11-09T18:05:15.866 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:15 +0000 (0:00:00.051) 0:02:07.592 ******* 2020-11-09T18:05:15.976 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "gid": 1234, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2020-11-09T18:05:16.020 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:16 +0000 (0:00:00.153) 0:02:07.746 ******* 2020-11-09T18:05:16.030 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"changed": false, "gid": 1234, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2020-11-09T18:05:16.066 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:16 +0000 (0:00:00.045) 0:02:07.792 ******* 2020-11-09T18:05:16.158 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2020-11-09T18:05:16.160 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2020-11-09T18:05:16.197 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:16 +0000 (0:00:00.129) 0:02:07.921 ******* 2020-11-09T18:05:16.212 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2020-11-09T18:05:16.249 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:16 +0000 (0:00:00.053) 0:02:07.975 ******* 2020-11-09T18:05:16.554 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2020-11-09T18:05:16.555 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1234, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2020-11-09T18:05:16.598 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:16 +0000 (0:00:00.348) 0:02:08.323 ******* 2020-11-09T18:05:16.607 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1234, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2020-11-09T18:05:16.644 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:16 +0000 (0:00:00.047) 0:02:08.370 ******* 2020-11-09T18:05:16.927 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2020-11-09T18:05:16.929 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2020-11-09T18:05:16.966 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:16 +0000 (0:00:00.321) 0:02:08.691 ******* 2020-11-09T18:05:16.998 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2020-11-09T18:05:17.036 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:17 +0000 (0:00:00.069) 0:02:08.761 ******* 2020-11-09T18:05:17.947 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2020-11-09T18:05:17.949 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"cache_update_time": 1604944975, "cache_updated": false, "changed": false} 2020-11-09T18:05:17.983 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:17 +0000 (0:00:00.947) 0:02:09.709 ******* 2020-11-09T18:05:17.990 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"cache_update_time": 1604944981, "cache_updated": false, "changed": false} 2020-11-09T18:05:18.030 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:18 +0000 (0:00:00.046) 0:02:09.756 ******* 2020-11-09T18:05:18.168 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* ok: [smithi051.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.048862", "end": "2020-11-09 18:05:17.809238", "rc": 0, "start": "2020-11-09 18:05:17.760376", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2020-11-09T18:05:18.202 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:18 +0000 (0:00:00.172) 0:02:09.928 ******* 2020-11-09T18:05:18.230 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.046702", "end": "2020-11-09 18:05:18.600136", "rc": 0, "start": "2020-11-09 18:05:18.553434", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2020-11-09T18:05:18.231 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:18.266 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:18 +0000 (0:00:00.063) 0:02:09.992 ******* 2020-11-09T18:05:18.304 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:18 +0000 (0:00:00.037) 0:02:10.030 ******* 2020-11-09T18:05:18.311 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2020-11-09T18:05:18.313 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:18.347 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:18 +0000 (0:00:00.043) 0:02:10.073 ******* 2020-11-09T18:05:18.354 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2020-11-09T18:05:18.356 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:18.390 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:18 +0000 (0:00:00.042) 0:02:10.116 ******* 2020-11-09T18:05:18.419 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:18.453 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:18 +0000 (0:00:00.063) 0:02:10.179 ******* 2020-11-09T18:05:19.224 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2020-11-09T18:05:19.226 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"cache_update_time": 1604944975, "cache_updated": false, "changed": false} 2020-11-09T18:05:19.259 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:19 +0000 (0:00:00.805) 0:02:10.985 ******* 2020-11-09T18:05:19.286 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2020-11-09T18:05:19.289 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:19.322 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:19 +0000 (0:00:00.062) 0:02:11.048 ******* 2020-11-09T18:05:19.341 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2020-11-09T18:05:19.343 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"cache_update_time": 1604944981, "cache_updated": false, "changed": false} 2020-11-09T18:05:19.386 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:19 +0000 (0:00:00.064) 0:02:11.112 ******* 2020-11-09T18:05:19.414 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2020-11-09T18:05:19.416 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:19.449 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:19 +0000 (0:00:00.062) 0:02:11.175 ******* 2020-11-09T18:05:19.656 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2020-11-09T18:05:19.657 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2020-11-09T18:05:19.697 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:19 +0000 (0:00:00.247) 0:02:11.422 ******* 2020-11-09T18:05:19.726 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2020-11-09T18:05:19.728 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:19.767 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:19 +0000 (0:00:00.070) 0:02:11.493 ******* 2020-11-09T18:05:19.775 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2020-11-09T18:05:19.777 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2020-11-09T18:05:19.815 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:19 +0000 (0:00:00.048) 0:02:11.541 ******* 2020-11-09T18:05:19.844 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2020-11-09T18:05:19.845 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:19.882 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:19 +0000 (0:00:00.065) 0:02:11.606 ******* 2020-11-09T18:05:19.985 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-11-09T18:05:19.986 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Mon 2020-11-09 18:01:49 UTC", "ActiveEnterTimestampMonotonic": "17129185", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-journald.socket basic.target systemd-tmpfiles-setup.service network.target system.slice -.mount sysinit.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2020-11-09 18:01:49 UTC", "AssertTimestampMonotonic": "17020125", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2020-11-09 18:01:49 UTC", "ConditionTimestampMonotonic": "17020125", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service shutdown.target", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2020-11-09T18:05:19.988 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1069", "ExecMainStartTimestamp": "Mon 2020-11-09 18:01:49 UTC", "ExecMainStartTimestampMonotonic": "17129169", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2020-11-09 18:01:49 UTC", "InactiveExitTimestampMonotonic": "17020996", "InvocationID": "7f7b4154ce804cd2831e1e9d22838890", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128055", "LimitNPROCSoft": "128055", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2020-11-09T18:05:19.989 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128055", "LimitSIGPENDINGSoft": "128055", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1069", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2020-11-09 18:01:49 UTC", "S 2020-11-09T18:05:19.991 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "17129185", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2020-11-09 18:01:49 UTC", "WatchdogTimestampMonotonic": "17129185", "WatchdogUSec": "0"}} 2020-11-09T18:05:20.022 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:20 +0000 (0:00:00.142) 0:02:11.748 ******* 2020-11-09T18:05:20.051 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-09T18:05:20.053 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2020-11-09T18:05:20.086 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:20 +0000 (0:00:00.063) 0:02:11.811 ******* 2020-11-09T18:05:20.099 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-11-09T18:05:20.101 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Mon 2020-11-09 18:02:24 UTC", "ActiveEnterTimestampMonotonic": "24036209", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-tmpfiles-setup.service network.target basic.target system.slice sysinit.target -.mount systemd-journald.socket", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2020-11-09 18:02:17 UTC", "AssertTimestampMonotonic": "17204158", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2020-11-09 18:02:17 UTC", "ConditionTimestampMonotonic": "17204158", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target systemd-timesyncd.service", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2020-11-09T18:05:20.102 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "2007", "ExecMainStartTimestamp": "Mon 2020-11-09 18:02:24 UTC", "ExecMainStartTimestampMonotonic": "24036196", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2020-11-09 18:02:17 UTC", "InactiveExitTimestampMonotonic": "17205049", "InvocationID": "9a4a78b5318e4fe1aa28dc9ed2d3b1d5", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128030", "LimitNPROCSoft": "128030", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2020-11-09T18:05:20.104 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128030", "LimitSIGPENDINGSoft": "128030", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "2007", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit.target", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2020-11-09 18:02:24 UTC", "S 2020-11-09T18:05:20.105 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "24036209", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2020-11-09 18:02:24 UTC", "WatchdogTimestampMonotonic": "24036207", "WatchdogUSec": "0"}} 2020-11-09T18:05:20.139 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:20 +0000 (0:00:00.053) 0:02:11.865 ******* 2020-11-09T18:05:20.145 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-09T18:05:20.147 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:20.179 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:20 +0000 (0:00:00.040) 0:02:11.905 ******* 2020-11-09T18:05:20.188 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-09T18:05:20.190 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2020-11-09T18:05:20.223 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:20 +0000 (0:00:00.042) 0:02:11.948 ******* 2020-11-09T18:05:20.230 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:20.265 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:20 +0000 (0:00:00.042) 0:02:11.991 ******* 2020-11-09T18:05:20.272 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-09T18:05:20.273 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:20.307 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:20 +0000 (0:00:00.042) 0:02:12.033 ******* 2020-11-09T18:05:20.335 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:20.371 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:20 +0000 (0:00:00.063) 0:02:12.096 ******* 2020-11-09T18:05:21.187 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2020-11-09T18:05:21.189 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"cache_update_time": 1604944975, "cache_updated": false, "changed": false} 2020-11-09T18:05:21.222 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:21 +0000 (0:00:00.851) 0:02:12.948 ******* 2020-11-09T18:05:21.240 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"cache_update_time": 1604944981, "cache_updated": false, "changed": false} 2020-11-09T18:05:21.280 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:21 +0000 (0:00:00.057) 0:02:13.006 ******* 2020-11-09T18:05:21.290 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2020-11-09T18:05:21.291 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:21.325 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:21 +0000 (0:00:00.045) 0:02:13.051 ******* 2020-11-09T18:05:21.332 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:21.367 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:21 +0000 (0:00:00.042) 0:02:13.093 ******* 2020-11-09T18:05:21.459 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2020-11-09T18:05:21.461 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"changed": false, "gid": 1234, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2020-11-09T18:05:21.495 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:21 +0000 (0:00:00.127) 0:02:13.220 ******* 2020-11-09T18:05:21.512 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "gid": 1234, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2020-11-09T18:05:21.556 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:21 +0000 (0:00:00.061) 0:02:13.282 ******* 2020-11-09T18:05:21.827 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** ok: [smithi051.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1234, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2020-11-09T18:05:21.863 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:21 +0000 (0:00:00.307) 0:02:13.589 ******* 2020-11-09T18:05:21.966 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1234, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2020-11-09T18:05:22.002 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:21 +0000 (0:00:00.138) 0:02:13.727 ******* 2020-11-09T18:05:22.907 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2020-11-09T18:05:22.908 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-11-09T18:05:22.946 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:22 +0000 (0:00:00.944) 0:02:14.672 ******* 2020-11-09T18:05:22.954 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-11-09T18:05:22.990 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:22 +0000 (0:00:00.043) 0:02:14.716 ******* 2020-11-09T18:05:23.491 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2020-11-09T18:05:23.493 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"changed": false} 2020-11-09T18:05:23.526 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:23 +0000 (0:00:00.536) 0:02:15.252 ******* 2020-11-09T18:05:23.547 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false} 2020-11-09T18:05:23.581 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:23 +0000 (0:00:00.055) 0:02:15.307 ******* 2020-11-09T18:05:23.588 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2020-11-09T18:05:23.590 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"ansible_facts": {"interface": "ens1f0"}, "changed": false} 2020-11-09T18:05:23.624 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:23 +0000 (0:00:00.042) 0:02:15.349 ******* 2020-11-09T18:05:23.631 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2020-11-09T18:05:23.671 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:23 +0000 (0:00:00.047) 0:02:15.397 ******* 2020-11-09T18:05:23.773 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2020-11-09T18:05:23.775 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1604944925.4122663, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1604944925.4082663, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 8056, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1604944925.4082663, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 67, "uid": 0, "version": "3017472389", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2020-11-09T18:05:23.930 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:23 +0000 (0:00:00.258) 0:02:15.656 ******* 2020-11-09T18:05:23.938 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1604944959.0575209, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1604944959.0535228, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 8056, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1604944959.0535228, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "3017472389", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2020-11-09T18:05:23.977 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:23 +0000 (0:00:00.047) 0:02:15.703 ******* 2020-11-09T18:05:24.200 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2020-11-09T18:05:24.202 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2020-11-09T18:05:24.237 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:24 +0000 (0:00:00.259) 0:02:15.963 ******* 2020-11-09T18:05:24.247 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2020-11-09T18:05:24.294 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:24 +0000 (0:00:00.057) 0:02:16.020 ******* 2020-11-09T18:05:24.301 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2020-11-09T18:05:24.302 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-11-09T18:05:24.335 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:24 +0000 (0:00:00.041) 0:02:16.061 ******* 2020-11-09T18:05:24.342 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-11-09T18:05:24.376 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:24 +0000 (0:00:00.041) 0:02:16.102 ******* 2020-11-09T18:05:24.384 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:24.419 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:24 +0000 (0:00:00.042) 0:02:16.145 ******* 2020-11-09T18:05:24.426 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:24.464 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:24 +0000 (0:00:00.044) 0:02:16.190 ******* 2020-11-09T18:05:24.546 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2020-11-09T18:05:24.548 INFO:teuthology.task.ansible.out:changed: [smithi051.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-11-09T18:05:24.583 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:24 +0000 (0:00:00.118) 0:02:16.309 ******* 2020-11-09T18:05:24.594 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-11-09T18:05:24.629 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:24 +0000 (0:00:00.046) 0:02:16.355 ******* 2020-11-09T18:05:24.711 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2020-11-09T18:05:24.712 INFO:teuthology.task.ansible.out:changed: [smithi051.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-11-09T18:05:24.746 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:24 +0000 (0:00:00.117) 0:02:16.472 ******* 2020-11-09T18:05:24.764 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-11-09T18:05:24.808 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:24 +0000 (0:00:00.061) 0:02:16.534 ******* 2020-11-09T18:05:24.815 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2020-11-09T18:05:24.817 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:24.853 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:24 +0000 (0:00:00.044) 0:02:16.579 ******* 2020-11-09T18:05:24.864 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:24.909 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:24 +0000 (0:00:00.055) 0:02:16.635 ******* 2020-11-09T18:05:24.916 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2020-11-09T18:05:24.918 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item=ens1f1) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False"} skipping: [smithi051.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi051.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi051.front.sepia.ceph.com] => (item=ens1f0) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False"} 2020-11-09T18:05:24.918 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2020-11-09T18:05:24.962 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:24 +0000 (0:00:00.053) 0:02:16.688 ******* 2020-11-09T18:05:24.976 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2020-11-09T18:05:24.980 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2020-11-09T18:05:24.981 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2020-11-09T18:05:24.982 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2020-11-09T18:05:24.982 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2020-11-09T18:05:25.026 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:25 +0000 (0:00:00.063) 0:02:16.752 ******* 2020-11-09T18:05:25.034 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2020-11-09T18:05:25.036 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi051.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi051.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi051.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:25.036 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result 2020-11-09T18:05:25.038 INFO:teuthology.task.ansible.out: was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:25.086 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:25 +0000 (0:00:00.060) 0:02:16.812 ******* 2020-11-09T18:05:25.099 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:25.101 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi078.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi078.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi078.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:25.149 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:25 +0000 (0:00:00.062) 0:02:16.875 ******* 2020-11-09T18:05:25.157 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-09T18:05:25.158 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:25.215 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:25 +0000 (0:00:00.065) 0:02:16.940 ******* 2020-11-09T18:05:25.223 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:25.269 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:25 +0000 (0:00:00.054) 0:02:16.995 ******* 2020-11-09T18:05:25.276 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2020-11-09T18:05:25.278 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:25.312 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:25 +0000 (0:00:00.043) 0:02:17.038 ******* 2020-11-09T18:05:25.319 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:25.359 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:25 +0000 (0:00:00.046) 0:02:17.084 ******* 2020-11-09T18:05:25.366 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2020-11-09T18:05:25.368 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:25.416 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:25 +0000 (0:00:00.057) 0:02:17.142 ******* 2020-11-09T18:05:25.425 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:25.469 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:25 +0000 (0:00:00.052) 0:02:17.195 ******* 2020-11-09T18:05:25.475 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2020-11-09T18:05:25.477 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:25.511 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:25 +0000 (0:00:00.042) 0:02:17.237 ******* 2020-11-09T18:05:25.520 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:25.557 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:25 +0000 (0:00:00.045) 0:02:17.282 ******* 2020-11-09T18:05:25.564 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2020-11-09T18:05:25.566 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:25.604 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:25 +0000 (0:00:00.047) 0:02:17.330 ******* 2020-11-09T18:05:25.611 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:25.657 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:25 +0000 (0:00:00.052) 0:02:17.383 ******* 2020-11-09T18:05:25.664 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-11-09T18:05:25.666 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:25.709 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:25 +0000 (0:00:00.051) 0:02:17.435 ******* 2020-11-09T18:05:25.717 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:25.760 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:25 +0000 (0:00:00.050) 0:02:17.485 ******* 2020-11-09T18:05:25.766 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2020-11-09T18:05:25.768 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:25.812 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:25 +0000 (0:00:00.052) 0:02:17.537 ******* 2020-11-09T18:05:25.820 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:25.866 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:25 +0000 (0:00:00.054) 0:02:17.591 ******* 2020-11-09T18:05:25.873 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-11-09T18:05:25.874 INFO:teuthology.task.ansible.out:skipping: [smithi051.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:25.908 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:25 +0000 (0:00:00.041) 0:02:17.633 ******* 2020-11-09T18:05:25.915 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T18:05:25.956 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:25 +0000 (0:00:00.047) 0:02:17.681 ******* 2020-11-09T18:05:26.066 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-11-09T18:05:26.068 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.010974", "end": "2020-11-09 18:05:26.040176", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-11-09 18:05:26.029202", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-09T18:05:26.130 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.008989", "end": "2020-11-09 18:05:26.488058", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-11-09 18:05:26.479069", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-09T18:05:26.193 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.006091", "end": "2020-11-09 18:05:26.170023", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-11-09 18:05:26.163932", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-09T18:05:26.251 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003919", "end": "2020-11-09 18:05:26.617266", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-11-09 18:05:26.613347", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-09T18:05:26.314 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005588", "end": "2020-11-09 18:05:26.289102", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-11-09 18:05:26.283514", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2020-11-09T18:05:26.368 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005251", "end": "2020-11-09 18:05:26.737821", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-11-09 18:05:26.732570", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2020-11-09T18:05:26.452 INFO:teuthology.task.ansible.out:ok: [smithi051.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.029581", "end": "2020-11-09 18:05:26.428549", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-11-09 18:05:26.398968", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-09T18:05:26.489 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:26 +0000 (0:00:00.533) 0:02:18.214 ******* 2020-11-09T18:05:26.517 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-11-09T18:05:26.518 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-11-09T18:05:26.519 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.030593", "end": "2020-11-09 18:05:26.888289", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-11-09 18:05:26.857696", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-09T18:05:26.556 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:26 +0000 (0:00:00.067) 0:02:18.282 ******* 2020-11-09T18:05:26.592 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:26 +0000 (0:00:00.035) 0:02:18.318 ******* 2020-11-09T18:05:26.623 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-11-09T18:05:26.673 INFO:teuthology.task.ansible.out:Monday 09 November 2020 18:05:26 +0000 (0:00:00.080) 0:02:18.399 ******* 2020-11-09T18:05:26.697 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2020-11-09T18:05:26.698 INFO:teuthology.task.ansible.out:changed: [smithi051.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2020-11-09T18:05:26.808 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2020-11-09T18:05:26.900 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2020-11-09T18:05:26.904 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2020-11-09T18:05:26.907 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched 2020-11-09T18:05:26.909 INFO:teuthology.task.ansible.out: PLAY RECAP ********************************************************************* smithi051.front.sepia.ceph.com : ok=114 changed=28 unreachable=0 failed=0 skipped=175 rescued=0 ignored=0 smithi078.front.sepia.ceph.com : ok=114 changed=28 unreachable=0 failed=0 skipped=175 rescued=0 ignored=0 Monday 09 November 2020 18:05:26 +0000 (0:00:00.235) 0:02:18.634 ******* =============================================================================== users : Create all admin users with sudo access. ----------------------- 17.85s users : Update authorized_keys using the keys repo --------------------- 16.91s testnode : Ensure packages are not present. ----------------------------- 9.45s testnode : Zap all non-root disks --------------------------------------- 8.28s testnode : Install packages --------------------------------------------- 7.54s users : Remove revoked users -------------------------------------------- 4.31s testnode : Update apt cache. -------------------------------------------- 2.79s common : Update apt cache ----------------------------------------------- 2.02s testnode : Install apt keys --------------------------------------------- 1.93s Gathering Facts --------------------------------------------------------- 1.71s testnode : Create logical volume(s) ------------------------------------- 1.48s users : Update authorized_keys for each user with literal keys ---------- 1.47s testnode : Blow away lingering OSD data and FSIDs ----------------------- 1.40s testnode : Upgrade packages --------------------------------------------- 0.97s testnode : Ensure perl-doc is installed on apt systems. ----------------- 0.95s testnode : Install packages via pip ------------------------------------- 0.94s testnode : Install python-pip on apt based systems. --------------------- 0.85s testnode : Install ntp package on apt based systems. -------------------- 0.81s testnode : Install python-apt ---------------- 2020-11-09T18:05:26.911 INFO:teuthology.task.ansible.out:--------------------------- 0.76s Gathering Facts --------------------------------------------------------- 0.75s 2020-11-09T18:05:27.272 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi051.front.sepia.ceph.com'), Remote(name='ubuntu@smithi078.front.sepia.ceph.com')] 2020-11-09T18:05:27.273 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-09T18:05:27.274 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi051.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T18:05:27.357 INFO:teuthology.orchestra.run.smithi051:> true 2020-11-09T18:05:27.563 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-09T18:05:27.564 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T18:05:27.643 INFO:teuthology.orchestra.run.smithi078:> true 2020-11-09T18:05:27.839 INFO:teuthology.run_tasks:Running task clock... 2020-11-09T18:05:27.851 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2020-11-09T18:05:27.851 INFO:teuthology.orchestra.run:Running command with timeout 360 2020-11-09T18:05:27.852 INFO:teuthology.orchestra.run.smithi051:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-11-09T18:05:27.908 INFO:teuthology.orchestra.run.smithi051.stdout: 9 Nov 18:05:27 ntpd[8726]: ntpd 4.2.8p10@1.3728-o (1): Starting 2020-11-09T18:05:27.909 INFO:teuthology.orchestra.run.smithi051.stdout: 9 Nov 18:05:27 ntpd[8726]: Command line: ntpd -gq 2020-11-09T18:05:27.911 INFO:teuthology.orchestra.run.smithi051.stdout: 9 Nov 18:05:27 ntpd[8726]: proto: precision = 0.126 usec (-23) 2020-11-09T18:05:27.911 INFO:teuthology.orchestra.run.smithi051.stdout: 9 Nov 18:05:27 ntpd[8726]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-11-09T18:05:27.911 INFO:teuthology.orchestra.run.smithi051.stdout: 9 Nov 18:05:27 ntpd[8726]: restrict ::: KOD does nothing without LIMITED. 2020-11-09T18:05:27.912 INFO:teuthology.orchestra.run.smithi051.stdout: 9 Nov 18:05:27 ntpd[8726]: Listen and drop on 0 v6wildcard [::]:123 2020-11-09T18:05:27.912 INFO:teuthology.orchestra.run.smithi051.stdout: 9 Nov 18:05:27 ntpd[8726]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2020-11-09T18:05:27.912 INFO:teuthology.orchestra.run.smithi051.stdout: 9 Nov 18:05:27 ntpd[8726]: Listen normally on 2 lo 127.0.0.1:123 2020-11-09T18:05:27.913 INFO:teuthology.orchestra.run.smithi051.stdout: 9 Nov 18:05:27 ntpd[8726]: Listen normally on 3 ens1f0 172.21.15.51:123 2020-11-09T18:05:27.913 INFO:teuthology.orchestra.run.smithi051.stdout: 9 Nov 18:05:27 ntpd[8726]: Listen normally on 4 lo [::1]:123 2020-11-09T18:05:27.913 INFO:teuthology.orchestra.run.smithi051.stdout: 9 Nov 18:05:27 ntpd[8726]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:15e2%3]:123 2020-11-09T18:05:27.913 INFO:teuthology.orchestra.run.smithi051.stdout: 9 Nov 18:05:27 ntpd[8726]: Listening on routing socket on fd #22 for interface updates 2020-11-09T18:05:27.914 INFO:teuthology.orchestra.run.smithi051.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-11-09T18:05:27.914 INFO:teuthology.orchestra.run.smithi051.stderr:restrict ::: KOD does nothing without LIMITED. 2020-11-09T18:05:28.909 INFO:teuthology.orchestra.run.smithi051.stderr: 9 Nov 18:05:28 ntpd[8726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-09T18:05:28.910 INFO:teuthology.orchestra.run.smithi051.stderr: 9 Nov 18:05:28 ntpd[8726]: can't open /var/log/ntpstats/rawstats.20201109: Permission denied 2020-11-09T18:05:28.910 INFO:teuthology.orchestra.run.smithi051.stderr: 9 Nov 18:05:28 ntpd[8726]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-09T18:05:28.910 INFO:teuthology.orchestra.run.smithi051.stderr: 9 Nov 18:05:28 ntpd[8726]: can't open /var/log/ntpstats/peerstats.20201109: Permission denied 2020-11-09T18:05:30.908 INFO:teuthology.orchestra.run.smithi051.stderr: 9 Nov 18:05:30 ntpd[8726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-09T18:05:30.909 INFO:teuthology.orchestra.run.smithi051.stderr: 9 Nov 18:05:30 ntpd[8726]: can't open /var/log/ntpstats/rawstats.20201109: Permission denied 2020-11-09T18:05:30.909 INFO:teuthology.orchestra.run.smithi051.stderr: 9 Nov 18:05:30 ntpd[8726]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-09T18:05:30.909 INFO:teuthology.orchestra.run.smithi051.stderr: 9 Nov 18:05:30 ntpd[8726]: can't open /var/log/ntpstats/peerstats.20201109: Permission denied 2020-11-09T18:05:31.909 INFO:teuthology.orchestra.run.smithi051.stderr: 9 Nov 18:05:31 ntpd[8726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-09T18:05:31.909 INFO:teuthology.orchestra.run.smithi051.stderr: 9 Nov 18:05:31 ntpd[8726]: can't open /var/log/ntpstats/rawstats.20201109: Permission denied 2020-11-09T18:05:31.909 INFO:teuthology.orchestra.run.smithi051.stderr: 9 Nov 18:05:31 ntpd[8726]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-09T18:05:31.909 INFO:teuthology.orchestra.run.smithi051.stderr: 9 Nov 18:05:31 ntpd[8726]: can't open /var/log/ntpstats/peerstats.20201109: Permission denied 2020-11-09T18:05:31.910 INFO:teuthology.orchestra.run.smithi051.stderr: 9 Nov 18:05:31 ntpd[8726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-09T18:05:31.910 INFO:teuthology.orchestra.run.smithi051.stderr: 9 Nov 18:05:31 ntpd[8726]: can't open /var/log/ntpstats/rawstats.20201109: Permission denied 2020-11-09T18:05:31.910 INFO:teuthology.orchestra.run.smithi051.stderr: 9 Nov 18:05:31 ntpd[8726]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-09T18:05:31.910 INFO:teuthology.orchestra.run.smithi051.stderr: 9 Nov 18:05:31 ntpd[8726]: can't open /var/log/ntpstats/peerstats.20201109: Permission denied 2020-11-09T18:05:32.911 INFO:teuthology.orchestra.run.smithi051.stderr: 9 Nov 18:05:32 ntpd[8726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-09T18:05:32.911 INFO:teuthology.orchestra.run.smithi051.stderr: 9 Nov 18:05:32 ntpd[8726]: can't open /var/log/ntpstats/rawstats.20201109: Permission denied 2020-11-09T18:05:32.911 INFO:teuthology.orchestra.run.smithi051.stderr: 9 Nov 18:05:32 ntpd[8726]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-09T18:05:32.911 INFO:teuthology.orchestra.run.smithi051.stderr: 9 Nov 18:05:32 ntpd[8726]: can't open /var/log/ntpstats/peerstats.20201109: Permission denied 2020-11-09T18:05:32.911 INFO:teuthology.orchestra.run.smithi051.stderr: 9 Nov 18:05:32 ntpd[8726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-09T18:05:32.911 INFO:teuthology.orchestra.run.smithi051.stderr: 9 Nov 18:05:32 ntpd[8726]: can't open /var/log/ntpstats/rawstats.20201109: Permission denied 2020-11-09T18:05:32.912 INFO:teuthology.orchestra.run.smithi051.stderr: 9 Nov 18:05:32 ntpd[8726]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-09T18:05:32.912 INFO:teuthology.orchestra.run.smithi051.stderr: 9 Nov 18:05:32 ntpd[8726]: can't open /var/log/ntpstats/peerstats.20201109: Permission denied 2020-11-09T18:05:33.909 INFO:teuthology.orchestra.run.smithi051.stderr: 9 Nov 18:05:33 ntpd[8726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-09T18:05:33.909 INFO:teuthology.orchestra.run.smithi051.stderr: 9 Nov 18:05:33 ntpd[8726]: can't open /var/log/ntpstats/rawstats.20201109: Permission denied 2020-11-09T18:05:33.909 INFO:teuthology.orchestra.run.smithi051.stderr: 9 Nov 18:05:33 ntpd[8726]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-09T18:05:33.909 INFO:teuthology.orchestra.run.smithi051.stderr: 9 Nov 18:05:33 ntpd[8726]: can't open /var/log/ntpstats/peerstats.20201109: Permission denied 2020-11-09T18:05:33.909 INFO:teuthology.orchestra.run.smithi051.stderr: 9 Nov 18:05:33 ntpd[8726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-09T18:05:33.910 INFO:teuthology.orchestra.run.smithi051.stderr: 9 Nov 18:05:33 ntpd[8726]: can't open /var/log/ntpstats/rawstats.20201109: Permission denied 2020-11-09T18:05:33.911 INFO:teuthology.orchestra.run.smithi051.stderr: 9 Nov 18:05:33 ntpd[8726]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-09T18:05:33.912 INFO:teuthology.orchestra.run.smithi051.stderr: 9 Nov 18:05:33 ntpd[8726]: can't open /var/log/ntpstats/peerstats.20201109: Permission denied 2020-11-09T18:05:34.909 INFO:teuthology.orchestra.run.smithi051.stderr: 9 Nov 18:05:34 ntpd[8726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-09T18:05:34.909 INFO:teuthology.orchestra.run.smithi051.stderr: 9 Nov 18:05:34 ntpd[8726]: can't open /var/log/ntpstats/rawstats.20201109: Permission denied 2020-11-09T18:05:34.909 INFO:teuthology.orchestra.run.smithi051.stderr: 9 Nov 18:05:34 ntpd[8726]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-09T18:05:34.909 INFO:teuthology.orchestra.run.smithi051.stderr: 9 Nov 18:05:34 ntpd[8726]: can't open /var/log/ntpstats/peerstats.20201109: Permission denied 2020-11-09T18:05:34.909 INFO:teuthology.orchestra.run.smithi051.stderr: 9 Nov 18:05:34 ntpd[8726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-09T18:05:34.910 INFO:teuthology.orchestra.run.smithi051.stderr: 9 Nov 18:05:34 ntpd[8726]: can't open /var/log/ntpstats/rawstats.20201109: Permission denied 2020-11-09T18:05:34.910 INFO:teuthology.orchestra.run.smithi051.stderr: 9 Nov 18:05:34 ntpd[8726]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-09T18:05:34.910 INFO:teuthology.orchestra.run.smithi051.stderr: 9 Nov 18:05:34 ntpd[8726]: can't open /var/log/ntpstats/peerstats.20201109: Permission denied 2020-11-09T18:05:34.910 INFO:teuthology.orchestra.run.smithi051.stderr: 9 Nov 18:05:34 ntpd[8726]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2020-11-09T18:05:34.910 INFO:teuthology.orchestra.run.smithi051.stderr: 9 Nov 18:05:34 ntpd[8726]: can't open /var/log/ntpstats/loopstats.20201109: Permission denied 2020-11-09T18:05:34.910 INFO:teuthology.orchestra.run.smithi051.stdout: 9 Nov 18:05:34 ntpd[8726]: ntpd: time slew -0.002564 s 2020-11-09T18:05:34.911 INFO:teuthology.orchestra.run.smithi051.stdout:ntpd: time slew -0.002564s 2020-11-09T18:05:34.957 INFO:teuthology.orchestra.run.smithi051.stdout: remote refid st t when poll reach delay offset jitter 2020-11-09T18:05:34.957 INFO:teuthology.orchestra.run.smithi051.stdout:============================================================================== 2020-11-09T18:05:34.958 INFO:teuthology.orchestra.run.smithi051.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-09T18:05:34.958 INFO:teuthology.orchestra.run.smithi051.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-09T18:05:34.958 INFO:teuthology.orchestra.run.smithi051.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-09T18:05:34.958 INFO:teuthology.orchestra.run.smithi051.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-09T18:05:34.958 INFO:teuthology.orchestra.run:Running command with timeout 360 2020-11-09T18:05:34.959 INFO:teuthology.orchestra.run.smithi078:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-11-09T18:05:35.015 INFO:teuthology.orchestra.run.smithi078.stdout: 9 Nov 18:05:35 ntpd[9028]: ntpd 4.2.8p10@1.3728-o (1): Starting 2020-11-09T18:05:35.016 INFO:teuthology.orchestra.run.smithi078.stdout: 9 Nov 18:05:35 ntpd[9028]: Command line: ntpd -gq 2020-11-09T18:05:35.016 INFO:teuthology.orchestra.run.smithi078.stdout: 9 Nov 18:05:35 ntpd[9028]: proto: precision = 0.103 usec (-23) 2020-11-09T18:05:35.017 INFO:teuthology.orchestra.run.smithi078.stdout: 9 Nov 18:05:35 ntpd[9028]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-11-09T18:05:35.017 INFO:teuthology.orchestra.run.smithi078.stdout: 9 Nov 18:05:35 ntpd[9028]: restrict ::: KOD does nothing without LIMITED. 2020-11-09T18:05:35.017 INFO:teuthology.orchestra.run.smithi078.stdout: 9 Nov 18:05:35 ntpd[9028]: Listen and drop on 0 v6wildcard [::]:123 2020-11-09T18:05:35.017 INFO:teuthology.orchestra.run.smithi078.stdout: 9 Nov 18:05:35 ntpd[9028]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2020-11-09T18:05:35.017 INFO:teuthology.orchestra.run.smithi078.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-11-09T18:05:35.017 INFO:teuthology.orchestra.run.smithi078.stderr:restrict ::: KOD does nothing without LIMITED. 2020-11-09T18:05:35.018 INFO:teuthology.orchestra.run.smithi078.stdout: 9 Nov 18:05:35 ntpd[9028]: Listen normally on 2 lo 127.0.0.1:123 2020-11-09T18:05:35.018 INFO:teuthology.orchestra.run.smithi078.stdout: 9 Nov 18:05:35 ntpd[9028]: Listen normally on 3 enp3s0f1 172.21.15.78:123 2020-11-09T18:05:35.018 INFO:teuthology.orchestra.run.smithi078.stdout: 9 Nov 18:05:35 ntpd[9028]: Listen normally on 4 lo [::1]:123 2020-11-09T18:05:35.019 INFO:teuthology.orchestra.run.smithi078.stdout: 9 Nov 18:05:35 ntpd[9028]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d4dd%5]:123 2020-11-09T18:05:35.019 INFO:teuthology.orchestra.run.smithi078.stdout: 9 Nov 18:05:35 ntpd[9028]: Listening on routing socket on fd #22 for interface updates 2020-11-09T18:05:36.018 INFO:teuthology.orchestra.run.smithi078.stderr: 9 Nov 18:05:36 ntpd[9028]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-09T18:05:36.018 INFO:teuthology.orchestra.run.smithi078.stderr: 9 Nov 18:05:36 ntpd[9028]: can't open /var/log/ntpstats/rawstats.20201109: Permission denied 2020-11-09T18:05:36.018 INFO:teuthology.orchestra.run.smithi078.stderr: 9 Nov 18:05:36 ntpd[9028]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-09T18:05:36.019 INFO:teuthology.orchestra.run.smithi078.stderr: 9 Nov 18:05:36 ntpd[9028]: can't open /var/log/ntpstats/peerstats.20201109: Permission denied 2020-11-09T18:05:37.017 INFO:teuthology.orchestra.run.smithi078.stderr: 9 Nov 18:05:37 ntpd[9028]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-09T18:05:37.018 INFO:teuthology.orchestra.run.smithi078.stderr: 9 Nov 18:05:37 ntpd[9028]: can't open /var/log/ntpstats/rawstats.20201109: Permission denied 2020-11-09T18:05:37.018 INFO:teuthology.orchestra.run.smithi078.stderr: 9 Nov 18:05:37 ntpd[9028]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-09T18:05:37.019 INFO:teuthology.orchestra.run.smithi078.stderr: 9 Nov 18:05:37 ntpd[9028]: can't open /var/log/ntpstats/peerstats.20201109: Permission denied 2020-11-09T18:05:37.019 INFO:teuthology.orchestra.run.smithi078.stderr: 9 Nov 18:05:37 ntpd[9028]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-09T18:05:37.019 INFO:teuthology.orchestra.run.smithi078.stderr: 9 Nov 18:05:37 ntpd[9028]: can't open /var/log/ntpstats/rawstats.20201109: Permission denied 2020-11-09T18:05:37.019 INFO:teuthology.orchestra.run.smithi078.stderr: 9 Nov 18:05:37 ntpd[9028]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-09T18:05:37.020 INFO:teuthology.orchestra.run.smithi078.stderr: 9 Nov 18:05:37 ntpd[9028]: can't open /var/log/ntpstats/peerstats.20201109: Permission denied 2020-11-09T18:05:38.017 INFO:teuthology.orchestra.run.smithi078.stderr: 9 Nov 18:05:38 ntpd[9028]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-09T18:05:38.018 INFO:teuthology.orchestra.run.smithi078.stderr: 9 Nov 18:05:38 ntpd[9028]: can't open /var/log/ntpstats/rawstats.20201109: Permission denied 2020-11-09T18:05:38.018 INFO:teuthology.orchestra.run.smithi078.stderr: 9 Nov 18:05:38 ntpd[9028]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-09T18:05:38.018 INFO:teuthology.orchestra.run.smithi078.stderr: 9 Nov 18:05:38 ntpd[9028]: can't open /var/log/ntpstats/peerstats.20201109: Permission denied 2020-11-09T18:05:39.018 INFO:teuthology.orchestra.run.smithi078.stderr: 9 Nov 18:05:39 ntpd[9028]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-09T18:05:39.018 INFO:teuthology.orchestra.run.smithi078.stderr: 9 Nov 18:05:39 ntpd[9028]: can't open /var/log/ntpstats/rawstats.20201109: Permission denied 2020-11-09T18:05:39.019 INFO:teuthology.orchestra.run.smithi078.stderr: 9 Nov 18:05:39 ntpd[9028]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-09T18:05:39.019 INFO:teuthology.orchestra.run.smithi078.stderr: 9 Nov 18:05:39 ntpd[9028]: can't open /var/log/ntpstats/peerstats.20201109: Permission denied 2020-11-09T18:05:39.019 INFO:teuthology.orchestra.run.smithi078.stderr: 9 Nov 18:05:39 ntpd[9028]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-09T18:05:39.019 INFO:teuthology.orchestra.run.smithi078.stderr: 9 Nov 18:05:39 ntpd[9028]: can't open /var/log/ntpstats/rawstats.20201109: Permission denied 2020-11-09T18:05:39.019 INFO:teuthology.orchestra.run.smithi078.stderr: 9 Nov 18:05:39 ntpd[9028]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-09T18:05:39.019 INFO:teuthology.orchestra.run.smithi078.stderr: 9 Nov 18:05:39 ntpd[9028]: can't open /var/log/ntpstats/peerstats.20201109: Permission denied 2020-11-09T18:05:40.019 INFO:teuthology.orchestra.run.smithi078.stderr: 9 Nov 18:05:40 ntpd[9028]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-09T18:05:40.019 INFO:teuthology.orchestra.run.smithi078.stderr: 9 Nov 18:05:40 ntpd[9028]: can't open /var/log/ntpstats/rawstats.20201109: Permission denied 2020-11-09T18:05:40.019 INFO:teuthology.orchestra.run.smithi078.stderr: 9 Nov 18:05:40 ntpd[9028]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-09T18:05:40.019 INFO:teuthology.orchestra.run.smithi078.stderr: 9 Nov 18:05:40 ntpd[9028]: can't open /var/log/ntpstats/peerstats.20201109: Permission denied 2020-11-09T18:05:40.019 INFO:teuthology.orchestra.run.smithi078.stderr: 9 Nov 18:05:40 ntpd[9028]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-09T18:05:40.020 INFO:teuthology.orchestra.run.smithi078.stderr: 9 Nov 18:05:40 ntpd[9028]: can't open /var/log/ntpstats/rawstats.20201109: Permission denied 2020-11-09T18:05:40.020 INFO:teuthology.orchestra.run.smithi078.stderr: 9 Nov 18:05:40 ntpd[9028]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-09T18:05:40.020 INFO:teuthology.orchestra.run.smithi078.stderr: 9 Nov 18:05:40 ntpd[9028]: can't open /var/log/ntpstats/peerstats.20201109: Permission denied 2020-11-09T18:05:41.019 INFO:teuthology.orchestra.run.smithi078.stderr: 9 Nov 18:05:41 ntpd[9028]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-09T18:05:41.019 INFO:teuthology.orchestra.run.smithi078.stderr: 9 Nov 18:05:41 ntpd[9028]: can't open /var/log/ntpstats/rawstats.20201109: Permission denied 2020-11-09T18:05:41.019 INFO:teuthology.orchestra.run.smithi078.stderr: 9 Nov 18:05:41 ntpd[9028]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-09T18:05:41.020 INFO:teuthology.orchestra.run.smithi078.stderr: 9 Nov 18:05:41 ntpd[9028]: can't open /var/log/ntpstats/peerstats.20201109: Permission denied 2020-11-09T18:05:41.020 INFO:teuthology.orchestra.run.smithi078.stderr: 9 Nov 18:05:41 ntpd[9028]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-09T18:05:41.020 INFO:teuthology.orchestra.run.smithi078.stderr: 9 Nov 18:05:41 ntpd[9028]: can't open /var/log/ntpstats/rawstats.20201109: Permission denied 2020-11-09T18:05:41.020 INFO:teuthology.orchestra.run.smithi078.stderr: 9 Nov 18:05:41 ntpd[9028]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-09T18:05:41.020 INFO:teuthology.orchestra.run.smithi078.stderr: 9 Nov 18:05:41 ntpd[9028]: can't open /var/log/ntpstats/peerstats.20201109: Permission denied 2020-11-09T18:05:42.019 INFO:teuthology.orchestra.run.smithi078.stderr: 9 Nov 18:05:42 ntpd[9028]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-09T18:05:42.020 INFO:teuthology.orchestra.run.smithi078.stderr: 9 Nov 18:05:42 ntpd[9028]: can't open /var/log/ntpstats/rawstats.20201109: Permission denied 2020-11-09T18:05:42.020 INFO:teuthology.orchestra.run.smithi078.stderr: 9 Nov 18:05:42 ntpd[9028]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-09T18:05:42.020 INFO:teuthology.orchestra.run.smithi078.stderr: 9 Nov 18:05:42 ntpd[9028]: can't open /var/log/ntpstats/peerstats.20201109: Permission denied 2020-11-09T18:05:42.020 INFO:teuthology.orchestra.run.smithi078.stderr: 9 Nov 18:05:42 ntpd[9028]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2020-11-09T18:05:42.020 INFO:teuthology.orchestra.run.smithi078.stderr: 9 Nov 18:05:42 ntpd[9028]: can't open /var/log/ntpstats/loopstats.20201109: Permission denied 2020-11-09T18:05:42.021 INFO:teuthology.orchestra.run.smithi078.stdout: 9 Nov 18:05:42 ntpd[9028]: ntpd: time set -0.387444 s 2020-11-09T18:05:42.021 INFO:teuthology.orchestra.run.smithi078.stdout:ntpd: time set -0.387444s 2020-11-09T18:05:42.147 INFO:teuthology.orchestra.run.smithi078.stdout: remote refid st t when poll reach delay offset jitter 2020-11-09T18:05:42.148 INFO:teuthology.orchestra.run.smithi078.stdout:============================================================================== 2020-11-09T18:05:42.150 INFO:teuthology.orchestra.run.smithi078.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-09T18:05:42.151 INFO:teuthology.orchestra.run.smithi078.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-09T18:05:42.151 INFO:teuthology.orchestra.run.smithi078.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-09T18:05:42.152 INFO:teuthology.orchestra.run.smithi078.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-09T18:05:42.153 INFO:teuthology.run_tasks:Running task install... 2020-11-09T18:05:42.164 DEBUG:teuthology.task.install:project ceph 2020-11-09T18:05:42.165 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': 'a0560c8e5c6c33c4655e9d2a8b022793ae57419f'}} 2020-11-09T18:05:42.165 DEBUG:teuthology.task.install:config {'sha1': 'a0560c8e5c6c33c4655e9d2a8b022793ae57419f'} 2020-11-09T18:05:42.165 INFO:teuthology.task.install:Using flavor: basic 2020-11-09T18:05:42.171 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2020-11-09T18:05:42.172 INFO:teuthology.task.install:extra packages: [] 2020-11-09T18:05:42.172 INFO:teuthology.orchestra.run.smithi051:> sudo lsb_release -is 2020-11-09T18:05:42.224 INFO:teuthology.orchestra.run.smithi051.stdout:Ubuntu 2020-11-09T18:05:42.229 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-11-09T18:05:42.230 INFO:teuthology.orchestra.run.smithi078:> sudo lsb_release -is 2020-11-09T18:05:42.230 INFO:teuthology.orchestra.run.smithi051:> sudo apt-key list | grep Ceph 2020-11-09T18:05:42.283 INFO:teuthology.orchestra.run.smithi078.stdout:Ubuntu 2020-11-09T18:05:42.288 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-11-09T18:05:42.288 INFO:teuthology.orchestra.run.smithi078:> sudo apt-key list | grep Ceph 2020-11-09T18:05:42.338 INFO:teuthology.orchestra.run.smithi051.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2020-11-09T18:05:42.376 INFO:teuthology.orchestra.run.smithi078.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2020-11-09T18:05:42.402 INFO:teuthology.orchestra.run.smithi051.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2020-11-09T18:05:42.403 INFO:teuthology.orchestra.run.smithi051.stdout:uid [ unknown] Ceph.com (release key) 2020-11-09T18:05:42.404 INFO:teuthology.orchestra.run.smithi051:> sudo lsb_release -is 2020-11-09T18:05:42.436 INFO:teuthology.orchestra.run.smithi078.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2020-11-09T18:05:42.436 INFO:teuthology.orchestra.run.smithi078.stdout:uid [ unknown] Ceph.com (release key) 2020-11-09T18:05:42.437 INFO:teuthology.orchestra.run.smithi078:> sudo lsb_release -is 2020-11-09T18:05:42.458 INFO:teuthology.orchestra.run.smithi051.stdout:Ubuntu 2020-11-09T18:05:42.463 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-11-09T18:05:42.463 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2020-11-09T18:05:42.464 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=a0560c8e5c6c33c4655e9d2a8b022793ae57419f 2020-11-09T18:05:42.488 INFO:teuthology.orchestra.run.smithi078.stdout:Ubuntu 2020-11-09T18:05:42.493 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-11-09T18:05:42.494 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2020-11-09T18:05:42.494 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=a0560c8e5c6c33c4655e9d2a8b022793ae57419f 2020-11-09T18:05:42.642 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default/ 2020-11-09T18:05:42.642 INFO:teuthology.task.install.deb:Package version is 16.0.0-7048-ga0560c8e-1bionic 2020-11-09T18:05:42.662 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default/ 2020-11-09T18:05:42.663 INFO:teuthology.task.install.deb:Package version is 16.0.0-7048-ga0560c8e-1bionic 2020-11-09T18:05:42.772 INFO:teuthology.orchestra.run.smithi051:> set -ex 2020-11-09T18:05:42.772 INFO:teuthology.orchestra.run.smithi051:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2020-11-09T18:05:42.777 INFO:teuthology.orchestra.run.smithi051.stderr:+ sudo dd of=/etc/apt/sources.list.d/ceph.list 2020-11-09T18:05:42.787 INFO:teuthology.orchestra.run.smithi051.stderr:0+1 records in 2020-11-09T18:05:42.787 INFO:teuthology.orchestra.run.smithi051.stderr:0+1 records out 2020-11-09T18:05:42.787 INFO:teuthology.orchestra.run.smithi051.stderr:164 bytes copied, 6.869e-05 s, 2.4 MB/s 2020-11-09T18:05:42.790 INFO:teuthology.orchestra.run.smithi078:> set -ex 2020-11-09T18:05:42.791 INFO:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2020-11-09T18:05:42.792 INFO:teuthology.orchestra.run.smithi051:> sudo apt-get update 2020-11-09T18:05:42.796 INFO:teuthology.orchestra.run.smithi078.stderr:+ sudo dd of=/etc/apt/sources.list.d/ceph.list 2020-11-09T18:05:42.805 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records in 2020-11-09T18:05:42.805 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records out 2020-11-09T18:05:42.806 INFO:teuthology.orchestra.run.smithi078.stderr:164 bytes copied, 4.3044e-05 s, 3.8 MB/s 2020-11-09T18:05:42.807 INFO:teuthology.orchestra.run.smithi078:> sudo apt-get update 2020-11-09T18:05:42.991 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-11-09T18:05:43.059 INFO:teuthology.orchestra.run.smithi051.stdout:Ign:1 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic InRelease 2020-11-09T18:05:43.082 INFO:teuthology.orchestra.run.smithi078.stdout:Ign:2 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic InRelease 2020-11-09T18:05:43.095 INFO:teuthology.orchestra.run.smithi051.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-11-09T18:05:43.095 INFO:teuthology.orchestra.run.smithi051.stdout:Hit:3 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-11-09T18:05:43.095 INFO:teuthology.orchestra.run.smithi051.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2020-11-09T18:05:43.104 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-11-09T18:05:43.105 INFO:teuthology.orchestra.run.smithi078.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2020-11-09T18:05:43.119 INFO:teuthology.orchestra.run.smithi051.stdout:Ign:5 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic Release.gpg 2020-11-09T18:05:43.131 INFO:teuthology.orchestra.run.smithi078.stdout:Ign:5 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic Release.gpg 2020-11-09T18:05:43.143 INFO:teuthology.orchestra.run.smithi051.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main i386 Packages [2,951 B] 2020-11-09T18:05:43.158 INFO:teuthology.orchestra.run.smithi078.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main i386 Packages [2,951 B] 2020-11-09T18:05:43.162 INFO:teuthology.orchestra.run.smithi051.stdout:Hit:7 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-11-09T18:05:43.184 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:7 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-11-09T18:05:43.197 INFO:teuthology.orchestra.run.smithi051.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 Packages [16.1 kB] 2020-11-09T18:05:43.214 INFO:teuthology.orchestra.run.smithi078.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 Packages [16.1 kB] 2020-11-09T18:05:43.250 INFO:teuthology.orchestra.run.smithi051.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-11-09T18:05:43.265 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-11-09T18:05:43.586 INFO:teuthology.orchestra.run.smithi078.stdout:Fetched 26.7 kB in 1s (46.4 kB/s) 2020-11-09T18:05:43.734 INFO:teuthology.orchestra.run.smithi051.stdout:Fetched 26.7 kB in 1s (36.5 kB/s) 2020-11-09T18:05:44.804 INFO:teuthology.orchestra.run.smithi078.stdout:Reading package lists... 2020-11-09T18:05:44.849 INFO:teuthology.orchestra.run.smithi078:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=16.0.0-7048-ga0560c8e-1bionic cephadm=16.0.0-7048-ga0560c8e-1bionic ceph-mds=16.0.0-7048-ga0560c8e-1bionic ceph-mgr=16.0.0-7048-ga0560c8e-1bionic ceph-common=16.0.0-7048-ga0560c8e-1bionic ceph-fuse=16.0.0-7048-ga0560c8e-1bionic ceph-test=16.0.0-7048-ga0560c8e-1bionic radosgw=16.0.0-7048-ga0560c8e-1bionic python3-rados=16.0.0-7048-ga0560c8e-1bionic python3-rgw=16.0.0-7048-ga0560c8e-1bionic python3-cephfs=16.0.0-7048-ga0560c8e-1bionic python3-rbd=16.0.0-7048-ga0560c8e-1bionic libcephfs2=16.0.0-7048-ga0560c8e-1bionic libcephfs-dev=16.0.0-7048-ga0560c8e-1bionic librados2=16.0.0-7048-ga0560c8e-1bionic librbd1=16.0.0-7048-ga0560c8e-1bionic rbd-fuse=16.0.0-7048-ga0560c8e-1bionic 2020-11-09T18:05:44.933 INFO:teuthology.orchestra.run.smithi078.stdout:Reading package lists... 2020-11-09T18:05:45.001 INFO:teuthology.orchestra.run.smithi051.stdout:Reading package lists... 2020-11-09T18:05:45.048 INFO:teuthology.orchestra.run.smithi051:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=16.0.0-7048-ga0560c8e-1bionic cephadm=16.0.0-7048-ga0560c8e-1bionic ceph-mds=16.0.0-7048-ga0560c8e-1bionic ceph-mgr=16.0.0-7048-ga0560c8e-1bionic ceph-common=16.0.0-7048-ga0560c8e-1bionic ceph-fuse=16.0.0-7048-ga0560c8e-1bionic ceph-test=16.0.0-7048-ga0560c8e-1bionic radosgw=16.0.0-7048-ga0560c8e-1bionic python3-rados=16.0.0-7048-ga0560c8e-1bionic python3-rgw=16.0.0-7048-ga0560c8e-1bionic python3-cephfs=16.0.0-7048-ga0560c8e-1bionic python3-rbd=16.0.0-7048-ga0560c8e-1bionic libcephfs2=16.0.0-7048-ga0560c8e-1bionic libcephfs-dev=16.0.0-7048-ga0560c8e-1bionic librados2=16.0.0-7048-ga0560c8e-1bionic librbd1=16.0.0-7048-ga0560c8e-1bionic rbd-fuse=16.0.0-7048-ga0560c8e-1bionic 2020-11-09T18:05:45.126 INFO:teuthology.orchestra.run.smithi051.stdout:Reading package lists... 2020-11-09T18:05:45.197 INFO:teuthology.orchestra.run.smithi078.stdout:Building dependency tree... 2020-11-09T18:05:45.198 INFO:teuthology.orchestra.run.smithi078.stdout:Reading state information... 2020-11-09T18:05:45.361 INFO:teuthology.orchestra.run.smithi051.stdout:Building dependency tree... 2020-11-09T18:05:45.362 INFO:teuthology.orchestra.run.smithi051.stdout:Reading state information... 2020-11-09T18:05:45.369 INFO:teuthology.orchestra.run.smithi078.stdout:The following additional packages will be installed: 2020-11-09T18:05:45.369 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2020-11-09T18:05:45.371 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd jq 2020-11-09T18:05:45.371 INFO:teuthology.orchestra.run.smithi078.stdout: libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2020-11-09T18:05:45.372 INFO:teuthology.orchestra.run.smithi078.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 nvme-cli 2020-11-09T18:05:45.372 INFO:teuthology.orchestra.run.smithi078.stdout: python-attr python-funcsigs python-pastedeploy-tpl python-pluggy python-py 2020-11-09T18:05:45.372 INFO:teuthology.orchestra.run.smithi078.stdout: python-pytest python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2020-11-09T18:05:45.372 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-argparse python3-ceph-common python3-cffi-backend 2020-11-09T18:05:45.373 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cherrypy3 python3-cryptography python3-cycler python3-dateutil 2020-11-09T18:05:45.373 INFO:teuthology.orchestra.run.smithi078.stdout: python3-decorator python3-fasteners python3-html5lib python3-httplib2 2020-11-09T18:05:45.373 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch 2020-11-09T18:05:45.373 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2020-11-09T18:05:45.374 INFO:teuthology.orchestra.run.smithi078.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2020-11-09T18:05:45.374 INFO:teuthology.orchestra.run.smithi078.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2020-11-09T18:05:45.374 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2020-11-09T18:05:45.375 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2020-11-09T18:05:45.375 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2020-11-09T18:05:45.375 INFO:teuthology.orchestra.run.smithi078.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2020-11-09T18:05:45.375 INFO:teuthology.orchestra.run.smithi078.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-11-09T18:05:45.376 INFO:teuthology.orchestra.run.smithi078.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-11-09T18:05:45.376 INFO:teuthology.orchestra.run.smithi078.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-11-09T18:05:45.376 INFO:teuthology.orchestra.run.smithi078.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-11-09T18:05:45.376 INFO:teuthology.orchestra.run.smithi078.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2020-11-09T18:05:45.377 INFO:teuthology.orchestra.run.smithi078.stdout:Suggested packages: 2020-11-09T18:05:45.378 INFO:teuthology.orchestra.run.smithi078.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2020-11-09T18:05:45.378 INFO:teuthology.orchestra.run.smithi078.stdout: python-pytest-xdist python-mock python-cryptography-doc 2020-11-09T18:05:45.378 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2020-11-09T18:05:45.378 INFO:teuthology.orchestra.run.smithi078.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2020-11-09T18:05:45.379 INFO:teuthology.orchestra.run.smithi078.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2020-11-09T18:05:45.379 INFO:teuthology.orchestra.run.smithi078.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2020-11-09T18:05:45.379 INFO:teuthology.orchestra.run.smithi078.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2020-11-09T18:05:45.380 INFO:teuthology.orchestra.run.smithi078.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2020-11-09T18:05:45.380 INFO:teuthology.orchestra.run.smithi078.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2020-11-09T18:05:45.380 INFO:teuthology.orchestra.run.smithi078.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2020-11-09T18:05:45.380 INFO:teuthology.orchestra.run.smithi078.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2020-11-09T18:05:45.381 INFO:teuthology.orchestra.run.smithi078.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2020-11-09T18:05:45.381 INFO:teuthology.orchestra.run.smithi078.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2020-11-09T18:05:45.381 INFO:teuthology.orchestra.run.smithi078.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2020-11-09T18:05:45.381 INFO:teuthology.orchestra.run.smithi078.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2020-11-09T18:05:45.477 INFO:teuthology.orchestra.run.smithi078.stdout:The following NEW packages will be installed: 2020-11-09T18:05:45.477 INFO:teuthology.orchestra.run.smithi078.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2020-11-09T18:05:45.477 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2020-11-09T18:05:45.478 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq 2020-11-09T18:05:45.478 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs-dev libcephfs2 libjq1 libleveldb1v5 liblttng-ust0 liboath0 2020-11-09T18:05:45.479 INFO:teuthology.orchestra.run.smithi078.stdout: libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 libsnappy1v5 2020-11-09T18:05:45.479 INFO:teuthology.orchestra.run.smithi078.stdout: libxslt1.1 nvme-cli python-attr python-funcsigs python-pastedeploy-tpl 2020-11-09T18:05:45.479 INFO:teuthology.orchestra.run.smithi078.stdout: python-pluggy python-py python-pytest python3-asn1crypto python3-attr 2020-11-09T18:05:45.479 INFO:teuthology.orchestra.run.smithi078.stdout: python3-bcrypt python3-bs4 python3-ceph-argparse python3-ceph-common 2020-11-09T18:05:45.479 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cephfs python3-cffi-backend python3-cherrypy3 python3-cryptography 2020-11-09T18:05:45.479 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cycler python3-dateutil python3-decorator python3-fasteners 2020-11-09T18:05:45.480 INFO:teuthology.orchestra.run.smithi078.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib 2020-11-09T18:05:45.480 INFO:teuthology.orchestra.run.smithi078.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes 2020-11-09T18:05:45.480 INFO:teuthology.orchestra.run.smithi078.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2020-11-09T18:05:45.480 INFO:teuthology.orchestra.run.smithi078.stdout: python3-matplotlib python3-monotonic python3-numpy python3-oauth2client 2020-11-09T18:05:45.480 INFO:teuthology.orchestra.run.smithi078.stdout: python3-olefile python3-openssl python3-paste python3-pastedeploy 2020-11-09T18:05:45.480 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pastescript python3-pecan python3-pil python3-pluggy 2020-11-09T18:05:45.480 INFO:teuthology.orchestra.run.smithi078.stdout: python3-prettytable python3-py python3-pyasn1 python3-pyasn1-modules 2020-11-09T18:05:45.480 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyinotify python3-pyparsing python3-pytest python3-rados python3-rbd 2020-11-09T18:05:45.481 INFO:teuthology.orchestra.run.smithi078.stdout: python3-repoze.lru python3-rgw python3-routes python3-rsa python3-scipy 2020-11-09T18:05:45.481 INFO:teuthology.orchestra.run.smithi078.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-11-09T18:05:45.481 INFO:teuthology.orchestra.run.smithi078.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-11-09T18:05:45.481 INFO:teuthology.orchestra.run.smithi078.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-11-09T18:05:45.481 INFO:teuthology.orchestra.run.smithi078.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-11-09T18:05:45.481 INFO:teuthology.orchestra.run.smithi078.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2020-11-09T18:05:45.482 INFO:teuthology.orchestra.run.smithi078.stdout:The following packages will be upgraded: 2020-11-09T18:05:45.482 INFO:teuthology.orchestra.run.smithi078.stdout: librados2 librbd1 2020-11-09T18:05:45.531 INFO:teuthology.orchestra.run.smithi051.stdout:The following additional packages will be installed: 2020-11-09T18:05:45.532 INFO:teuthology.orchestra.run.smithi051.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2020-11-09T18:05:45.534 INFO:teuthology.orchestra.run.smithi051.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd jq 2020-11-09T18:05:45.534 INFO:teuthology.orchestra.run.smithi051.stdout: libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2020-11-09T18:05:45.535 INFO:teuthology.orchestra.run.smithi051.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 nvme-cli 2020-11-09T18:05:45.535 INFO:teuthology.orchestra.run.smithi051.stdout: python-attr python-funcsigs python-pastedeploy-tpl python-pluggy python-py 2020-11-09T18:05:45.535 INFO:teuthology.orchestra.run.smithi051.stdout: python-pytest python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2020-11-09T18:05:45.535 INFO:teuthology.orchestra.run.smithi051.stdout: python3-ceph-argparse python3-ceph-common python3-cffi-backend 2020-11-09T18:05:45.536 INFO:teuthology.orchestra.run.smithi051.stdout: python3-cherrypy3 python3-cryptography python3-cycler python3-dateutil 2020-11-09T18:05:45.536 INFO:teuthology.orchestra.run.smithi051.stdout: python3-decorator python3-fasteners python3-html5lib python3-httplib2 2020-11-09T18:05:45.536 INFO:teuthology.orchestra.run.smithi051.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch 2020-11-09T18:05:45.536 INFO:teuthology.orchestra.run.smithi051.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2020-11-09T18:05:45.537 INFO:teuthology.orchestra.run.smithi051.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2020-11-09T18:05:45.537 INFO:teuthology.orchestra.run.smithi051.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2020-11-09T18:05:45.537 INFO:teuthology.orchestra.run.smithi051.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2020-11-09T18:05:45.537 INFO:teuthology.orchestra.run.smithi051.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2020-11-09T18:05:45.537 INFO:teuthology.orchestra.run.smithi051.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2020-11-09T18:05:45.538 INFO:teuthology.orchestra.run.smithi051.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2020-11-09T18:05:45.538 INFO:teuthology.orchestra.run.smithi051.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-11-09T18:05:45.538 INFO:teuthology.orchestra.run.smithi051.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-11-09T18:05:45.538 INFO:teuthology.orchestra.run.smithi051.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-11-09T18:05:45.539 INFO:teuthology.orchestra.run.smithi051.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-11-09T18:05:45.539 INFO:teuthology.orchestra.run.smithi051.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2020-11-09T18:05:45.540 INFO:teuthology.orchestra.run.smithi051.stdout:Suggested packages: 2020-11-09T18:05:45.540 INFO:teuthology.orchestra.run.smithi051.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2020-11-09T18:05:45.540 INFO:teuthology.orchestra.run.smithi051.stdout: python-pytest-xdist python-mock python-cryptography-doc 2020-11-09T18:05:45.541 INFO:teuthology.orchestra.run.smithi051.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2020-11-09T18:05:45.541 INFO:teuthology.orchestra.run.smithi051.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2020-11-09T18:05:45.541 INFO:teuthology.orchestra.run.smithi051.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2020-11-09T18:05:45.542 INFO:teuthology.orchestra.run.smithi051.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2020-11-09T18:05:45.542 INFO:teuthology.orchestra.run.smithi051.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2020-11-09T18:05:45.542 INFO:teuthology.orchestra.run.smithi051.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2020-11-09T18:05:45.543 INFO:teuthology.orchestra.run.smithi051.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2020-11-09T18:05:45.543 INFO:teuthology.orchestra.run.smithi051.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2020-11-09T18:05:45.543 INFO:teuthology.orchestra.run.smithi051.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2020-11-09T18:05:45.543 INFO:teuthology.orchestra.run.smithi051.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2020-11-09T18:05:45.544 INFO:teuthology.orchestra.run.smithi051.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2020-11-09T18:05:45.544 INFO:teuthology.orchestra.run.smithi051.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2020-11-09T18:05:45.544 INFO:teuthology.orchestra.run.smithi051.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2020-11-09T18:05:45.638 INFO:teuthology.orchestra.run.smithi051.stdout:The following NEW packages will be installed: 2020-11-09T18:05:45.639 INFO:teuthology.orchestra.run.smithi051.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2020-11-09T18:05:45.639 INFO:teuthology.orchestra.run.smithi051.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2020-11-09T18:05:45.639 INFO:teuthology.orchestra.run.smithi051.stdout: ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq 2020-11-09T18:05:45.640 INFO:teuthology.orchestra.run.smithi051.stdout: libcephfs-dev libcephfs2 libjq1 libleveldb1v5 liblttng-ust0 liboath0 2020-11-09T18:05:45.641 INFO:teuthology.orchestra.run.smithi051.stdout: libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 libsnappy1v5 2020-11-09T18:05:45.641 INFO:teuthology.orchestra.run.smithi051.stdout: libxslt1.1 nvme-cli python-attr python-funcsigs python-pastedeploy-tpl 2020-11-09T18:05:45.641 INFO:teuthology.orchestra.run.smithi051.stdout: python-pluggy python-py python-pytest python3-asn1crypto python3-attr 2020-11-09T18:05:45.641 INFO:teuthology.orchestra.run.smithi051.stdout: python3-bcrypt python3-bs4 python3-ceph-argparse python3-ceph-common 2020-11-09T18:05:45.641 INFO:teuthology.orchestra.run.smithi051.stdout: python3-cephfs python3-cffi-backend python3-cherrypy3 python3-cryptography 2020-11-09T18:05:45.641 INFO:teuthology.orchestra.run.smithi051.stdout: python3-cycler python3-dateutil python3-decorator python3-fasteners 2020-11-09T18:05:45.642 INFO:teuthology.orchestra.run.smithi051.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib 2020-11-09T18:05:45.642 INFO:teuthology.orchestra.run.smithi051.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes 2020-11-09T18:05:45.642 INFO:teuthology.orchestra.run.smithi051.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2020-11-09T18:05:45.642 INFO:teuthology.orchestra.run.smithi051.stdout: python3-matplotlib python3-monotonic python3-numpy python3-oauth2client 2020-11-09T18:05:45.642 INFO:teuthology.orchestra.run.smithi051.stdout: python3-olefile python3-openssl python3-paste python3-pastedeploy 2020-11-09T18:05:45.642 INFO:teuthology.orchestra.run.smithi051.stdout: python3-pastescript python3-pecan python3-pil python3-pluggy 2020-11-09T18:05:45.643 INFO:teuthology.orchestra.run.smithi051.stdout: python3-prettytable python3-py python3-pyasn1 python3-pyasn1-modules 2020-11-09T18:05:45.643 INFO:teuthology.orchestra.run.smithi051.stdout: python3-pyinotify python3-pyparsing python3-pytest python3-rados python3-rbd 2020-11-09T18:05:45.643 INFO:teuthology.orchestra.run.smithi051.stdout: python3-repoze.lru python3-rgw python3-routes python3-rsa python3-scipy 2020-11-09T18:05:45.643 INFO:teuthology.orchestra.run.smithi051.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-11-09T18:05:45.643 INFO:teuthology.orchestra.run.smithi051.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-11-09T18:05:45.643 INFO:teuthology.orchestra.run.smithi051.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-11-09T18:05:45.643 INFO:teuthology.orchestra.run.smithi051.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-11-09T18:05:45.644 INFO:teuthology.orchestra.run.smithi051.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2020-11-09T18:05:45.644 INFO:teuthology.orchestra.run.smithi078.stdout:2 upgraded, 110 newly installed, 0 to remove and 2 not upgraded. 2020-11-09T18:05:45.644 INFO:teuthology.orchestra.run.smithi078.stdout:Need to get 168 MB of archives. 2020-11-09T18:05:45.645 INFO:teuthology.orchestra.run.smithi078.stdout:After this operation, 781 MB of additional disk space will be used. 2020-11-09T18:05:45.645 INFO:teuthology.orchestra.run.smithi078.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 16.0.0-7048-ga0560c8e-1bionic [1,920 kB] 2020-11-09T18:05:45.645 INFO:teuthology.orchestra.run.smithi051.stdout:The following packages will be upgraded: 2020-11-09T18:05:45.645 INFO:teuthology.orchestra.run.smithi051.stdout: librados2 librbd1 2020-11-09T18:05:45.671 INFO:teuthology.orchestra.run.smithi078.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2020-11-09T18:05:45.751 INFO:teuthology.orchestra.run.smithi078.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2020-11-09T18:05:45.809 INFO:teuthology.orchestra.run.smithi051.stdout:2 upgraded, 110 newly installed, 0 to remove and 2 not upgraded. 2020-11-09T18:05:45.809 INFO:teuthology.orchestra.run.smithi051.stdout:Need to get 168 MB of archives. 2020-11-09T18:05:45.810 INFO:teuthology.orchestra.run.smithi051.stdout:After this operation, 781 MB of additional disk space will be used. 2020-11-09T18:05:45.810 INFO:teuthology.orchestra.run.smithi051.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 16.0.0-7048-ga0560c8e-1bionic [1,920 kB] 2020-11-09T18:05:45.831 INFO:teuthology.orchestra.run.smithi051.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2020-11-09T18:05:45.837 INFO:teuthology.orchestra.run.smithi078.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 16.0.0-7048-ga0560c8e-1bionic [3,156 kB] 2020-11-09T18:05:45.912 INFO:teuthology.orchestra.run.smithi051.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2020-11-09T18:05:45.944 INFO:teuthology.orchestra.run.smithi078.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 16.0.0-7048-ga0560c8e-1bionic [478 kB] 2020-11-09T18:05:45.959 INFO:teuthology.orchestra.run.smithi078.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 16.0.0-7048-ga0560c8e-1bionic [331 kB] 2020-11-09T18:05:45.970 INFO:teuthology.orchestra.run.smithi078.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 16.0.0-7048-ga0560c8e-1bionic [20.6 kB] 2020-11-09T18:05:45.971 INFO:teuthology.orchestra.run.smithi078.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 16.0.0-7048-ga0560c8e-1bionic [168 kB] 2020-11-09T18:05:45.976 INFO:teuthology.orchestra.run.smithi078.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 16.0.0-7048-ga0560c8e-1bionic [27.7 kB] 2020-11-09T18:05:45.977 INFO:teuthology.orchestra.run.smithi078.stdout:Get:10 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 16.0.0-7048-ga0560c8e-1bionic [330 kB] 2020-11-09T18:05:45.993 INFO:teuthology.orchestra.run.smithi078.stdout:Get:11 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 16.0.0-7048-ga0560c8e-1bionic [3,169 kB] 2020-11-09T18:05:46.004 INFO:teuthology.orchestra.run.smithi078.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2020-11-09T18:05:46.018 INFO:teuthology.orchestra.run.smithi078.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2020-11-09T18:05:46.020 INFO:teuthology.orchestra.run.smithi078.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2020-11-09T18:05:46.108 INFO:teuthology.orchestra.run.smithi078.stdout:Get:15 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 16.0.0-7048-ga0560c8e-1bionic [99.1 kB] 2020-11-09T18:05:46.112 INFO:teuthology.orchestra.run.smithi078.stdout:Get:16 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 16.0.0-7048-ga0560c8e-1bionic [414 kB] 2020-11-09T18:05:46.129 INFO:teuthology.orchestra.run.smithi078.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2020-11-09T18:05:46.130 INFO:teuthology.orchestra.run.smithi078.stdout:Get:18 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 16.0.0-7048-ga0560c8e-1bionic [20.5 MB] 2020-11-09T18:05:46.131 INFO:teuthology.orchestra.run.smithi078.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2020-11-09T18:05:46.165 INFO:teuthology.orchestra.run.smithi078.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2020-11-09T18:05:46.166 INFO:teuthology.orchestra.run.smithi051.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2020-11-09T18:05:46.172 INFO:teuthology.orchestra.run.smithi078.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2020-11-09T18:05:46.180 INFO:teuthology.orchestra.run.smithi078.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2020-11-09T18:05:46.181 INFO:teuthology.orchestra.run.smithi051.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2020-11-09T18:05:46.191 INFO:teuthology.orchestra.run.smithi078.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2020-11-09T18:05:46.196 INFO:teuthology.orchestra.run.smithi051.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2020-11-09T18:05:46.201 INFO:teuthology.orchestra.run.smithi078.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.4 [220 kB] 2020-11-09T18:05:46.226 INFO:teuthology.orchestra.run.smithi078.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2020-11-09T18:05:46.230 INFO:teuthology.orchestra.run.smithi078.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2020-11-09T18:05:46.246 INFO:teuthology.orchestra.run.smithi078.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2020-11-09T18:05:46.267 INFO:teuthology.orchestra.run.smithi078.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2020-11-09T18:05:46.268 INFO:teuthology.orchestra.run.smithi078.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2020-11-09T18:05:46.295 INFO:teuthology.orchestra.run.smithi051.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2020-11-09T18:05:46.298 INFO:teuthology.orchestra.run.smithi051.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2020-11-09T18:05:46.324 INFO:teuthology.orchestra.run.smithi051.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2020-11-09T18:05:46.326 INFO:teuthology.orchestra.run.smithi078.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2020-11-09T18:05:46.326 INFO:teuthology.orchestra.run.smithi078.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2020-11-09T18:05:46.329 INFO:teuthology.orchestra.run.smithi078.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2020-11-09T18:05:46.330 INFO:teuthology.orchestra.run.smithi078.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2020-11-09T18:05:46.330 INFO:teuthology.orchestra.run.smithi078.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2020-11-09T18:05:46.331 INFO:teuthology.orchestra.run.smithi051.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2020-11-09T18:05:46.332 INFO:teuthology.orchestra.run.smithi078.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2020-11-09T18:05:46.334 INFO:teuthology.orchestra.run.smithi078.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2020-11-09T18:05:46.341 INFO:teuthology.orchestra.run.smithi051.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2020-11-09T18:05:46.352 INFO:teuthology.orchestra.run.smithi051.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2020-11-09T18:05:46.362 INFO:teuthology.orchestra.run.smithi051.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.4 [220 kB] 2020-11-09T18:05:46.386 INFO:teuthology.orchestra.run.smithi051.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2020-11-09T18:05:46.389 INFO:teuthology.orchestra.run.smithi051.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2020-11-09T18:05:46.406 INFO:teuthology.orchestra.run.smithi078.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2020-11-09T18:05:46.407 INFO:teuthology.orchestra.run.smithi078.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2020-11-09T18:05:46.408 INFO:teuthology.orchestra.run.smithi051.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2020-11-09T18:05:46.420 INFO:teuthology.orchestra.run.smithi051.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2020-11-09T18:05:46.421 INFO:teuthology.orchestra.run.smithi051.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2020-11-09T18:05:46.421 INFO:teuthology.orchestra.run.smithi078.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2020-11-09T18:05:46.468 INFO:teuthology.orchestra.run.smithi051.stdout:Get:19 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 16.0.0-7048-ga0560c8e-1bionic [3,156 kB] 2020-11-09T18:05:46.485 INFO:teuthology.orchestra.run.smithi078.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2020-11-09T18:05:46.486 INFO:teuthology.orchestra.run.smithi078.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2020-11-09T18:05:46.487 INFO:teuthology.orchestra.run.smithi051.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2020-11-09T18:05:46.488 INFO:teuthology.orchestra.run.smithi078.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2020-11-09T18:05:46.488 INFO:teuthology.orchestra.run.smithi051.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2020-11-09T18:05:46.490 INFO:teuthology.orchestra.run.smithi078.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1 [174 kB] 2020-11-09T18:05:46.490 INFO:teuthology.orchestra.run.smithi051.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2020-11-09T18:05:46.491 INFO:teuthology.orchestra.run.smithi051.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2020-11-09T18:05:46.492 INFO:teuthology.orchestra.run.smithi051.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2020-11-09T18:05:46.494 INFO:teuthology.orchestra.run.smithi051.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2020-11-09T18:05:46.495 INFO:teuthology.orchestra.run.smithi078.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2020-11-09T18:05:46.497 INFO:teuthology.orchestra.run.smithi051.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2020-11-09T18:05:46.498 INFO:teuthology.orchestra.run.smithi078.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2020-11-09T18:05:46.498 INFO:teuthology.orchestra.run.smithi078.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2020-11-09T18:05:46.566 INFO:teuthology.orchestra.run.smithi078.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2020-11-09T18:05:46.567 INFO:teuthology.orchestra.run.smithi051.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2020-11-09T18:05:46.582 INFO:teuthology.orchestra.run.smithi051.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2020-11-09T18:05:46.602 INFO:teuthology.orchestra.run.smithi051.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2020-11-09T18:05:46.647 INFO:teuthology.orchestra.run.smithi051.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2020-11-09T18:05:46.647 INFO:teuthology.orchestra.run.smithi051.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2020-11-09T18:05:46.650 INFO:teuthology.orchestra.run.smithi051.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2020-11-09T18:05:46.652 INFO:teuthology.orchestra.run.smithi051.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1 [174 kB] 2020-11-09T18:05:46.661 INFO:teuthology.orchestra.run.smithi051.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2020-11-09T18:05:46.665 INFO:teuthology.orchestra.run.smithi051.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2020-11-09T18:05:46.666 INFO:teuthology.orchestra.run.smithi051.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2020-11-09T18:05:46.675 INFO:teuthology.orchestra.run.smithi078.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2020-11-09T18:05:46.675 INFO:teuthology.orchestra.run.smithi078.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2020-11-09T18:05:46.728 INFO:teuthology.orchestra.run.smithi051.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2020-11-09T18:05:46.773 INFO:teuthology.orchestra.run.smithi051.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2020-11-09T18:05:46.773 INFO:teuthology.orchestra.run.smithi051.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2020-11-09T18:05:46.923 INFO:teuthology.orchestra.run.smithi078.stdout:Get:50 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 16.0.0-7048-ga0560c8e-1bionic [5,566 kB] 2020-11-09T18:05:46.956 INFO:teuthology.orchestra.run.smithi078.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2020-11-09T18:05:47.003 INFO:teuthology.orchestra.run.smithi078.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2020-11-09T18:05:47.005 INFO:teuthology.orchestra.run.smithi078.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2020-11-09T18:05:47.038 INFO:teuthology.orchestra.run.smithi078.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2020-11-09T18:05:47.039 INFO:teuthology.orchestra.run.smithi078.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2020-11-09T18:05:47.039 INFO:teuthology.orchestra.run.smithi078.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.2 [28.6 kB] 2020-11-09T18:05:47.040 INFO:teuthology.orchestra.run.smithi078.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2020-11-09T18:05:47.041 INFO:teuthology.orchestra.run.smithi078.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2020-11-09T18:05:47.042 INFO:teuthology.orchestra.run.smithi078.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2020-11-09T18:05:47.043 INFO:teuthology.orchestra.run.smithi078.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2020-11-09T18:05:47.070 INFO:teuthology.orchestra.run.smithi051.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2020-11-09T18:05:47.097 INFO:teuthology.orchestra.run.smithi051.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2020-11-09T18:05:47.107 INFO:teuthology.orchestra.run.smithi078.stdout:Get:61 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 16.0.0-7048-ga0560c8e-1bionic [165 kB] 2020-11-09T18:05:47.112 INFO:teuthology.orchestra.run.smithi078.stdout:Get:62 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 16.0.0-7048-ga0560c8e-1bionic [1,361 kB] 2020-11-09T18:05:47.117 INFO:teuthology.orchestra.run.smithi078.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2020-11-09T18:05:47.119 INFO:teuthology.orchestra.run.smithi078.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2020-11-09T18:05:47.125 INFO:teuthology.orchestra.run.smithi051.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2020-11-09T18:05:47.152 INFO:teuthology.orchestra.run.smithi051.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2020-11-09T18:05:47.152 INFO:teuthology.orchestra.run.smithi051.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2020-11-09T18:05:47.153 INFO:teuthology.orchestra.run.smithi051.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.2 [28.6 kB] 2020-11-09T18:05:47.153 INFO:teuthology.orchestra.run.smithi051.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2020-11-09T18:05:47.153 INFO:teuthology.orchestra.run.smithi051.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2020-11-09T18:05:47.154 INFO:teuthology.orchestra.run.smithi051.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2020-11-09T18:05:47.154 INFO:teuthology.orchestra.run.smithi051.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2020-11-09T18:05:47.164 INFO:teuthology.orchestra.run.smithi078.stdout:Get:65 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 16.0.0-7048-ga0560c8e-1bionic [6,652 kB] 2020-11-09T18:05:47.197 INFO:teuthology.orchestra.run.smithi078.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2020-11-09T18:05:47.199 INFO:teuthology.orchestra.run.smithi078.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2020-11-09T18:05:47.205 INFO:teuthology.orchestra.run.smithi078.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2020-11-09T18:05:47.205 INFO:teuthology.orchestra.run.smithi078.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2020-11-09T18:05:47.206 INFO:teuthology.orchestra.run.smithi078.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2020-11-09T18:05:47.207 INFO:teuthology.orchestra.run.smithi078.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2020-11-09T18:05:47.209 INFO:teuthology.orchestra.run.smithi078.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2020-11-09T18:05:47.210 INFO:teuthology.orchestra.run.smithi078.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2020-11-09T18:05:47.230 INFO:teuthology.orchestra.run.smithi051.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2020-11-09T18:05:47.232 INFO:teuthology.orchestra.run.smithi051.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2020-11-09T18:05:47.277 INFO:teuthology.orchestra.run.smithi078.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2020-11-09T18:05:47.282 INFO:teuthology.orchestra.run.smithi078.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2020-11-09T18:05:47.310 INFO:teuthology.orchestra.run.smithi051.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2020-11-09T18:05:47.310 INFO:teuthology.orchestra.run.smithi051.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2020-11-09T18:05:47.314 INFO:teuthology.orchestra.run.smithi051.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2020-11-09T18:05:47.314 INFO:teuthology.orchestra.run.smithi051.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2020-11-09T18:05:47.315 INFO:teuthology.orchestra.run.smithi051.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2020-11-09T18:05:47.317 INFO:teuthology.orchestra.run.smithi051.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2020-11-09T18:05:47.319 INFO:teuthology.orchestra.run.smithi051.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2020-11-09T18:05:47.320 INFO:teuthology.orchestra.run.smithi051.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2020-11-09T18:05:47.357 INFO:teuthology.orchestra.run.smithi078.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 nvme-cli amd64 1.5-1ubuntu1 [183 kB] 2020-11-09T18:05:47.361 INFO:teuthology.orchestra.run.smithi078.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2020-11-09T18:05:47.362 INFO:teuthology.orchestra.run.smithi078.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2020-11-09T18:05:47.362 INFO:teuthology.orchestra.run.smithi078.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2020-11-09T18:05:47.363 INFO:teuthology.orchestra.run.smithi078.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2020-11-09T18:05:47.364 INFO:teuthology.orchestra.run.smithi078.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2020-11-09T18:05:47.366 INFO:teuthology.orchestra.run.smithi078.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2020-11-09T18:05:47.367 INFO:teuthology.orchestra.run.smithi078.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2020-11-09T18:05:47.369 INFO:teuthology.orchestra.run.smithi051.stdout:Get:60 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 16.0.0-7048-ga0560c8e-1bionic [478 kB] 2020-11-09T18:05:47.389 INFO:teuthology.orchestra.run.smithi051.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2020-11-09T18:05:47.395 INFO:teuthology.orchestra.run.smithi051.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2020-11-09T18:05:47.414 INFO:teuthology.orchestra.run.smithi078.stdout:Get:84 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 16.0.0-7048-ga0560c8e-1bionic [24.6 MB] 2020-11-09T18:05:47.436 INFO:teuthology.orchestra.run.smithi078.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2020-11-09T18:05:47.437 INFO:teuthology.orchestra.run.smithi078.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2020-11-09T18:05:47.469 INFO:teuthology.orchestra.run.smithi051.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 nvme-cli amd64 1.5-1ubuntu1 [183 kB] 2020-11-09T18:05:47.476 INFO:teuthology.orchestra.run.smithi051.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2020-11-09T18:05:47.476 INFO:teuthology.orchestra.run.smithi051.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2020-11-09T18:05:47.477 INFO:teuthology.orchestra.run.smithi051.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2020-11-09T18:05:47.478 INFO:teuthology.orchestra.run.smithi051.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2020-11-09T18:05:47.480 INFO:teuthology.orchestra.run.smithi051.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2020-11-09T18:05:47.481 INFO:teuthology.orchestra.run.smithi051.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2020-11-09T18:05:47.482 INFO:teuthology.orchestra.run.smithi051.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2020-11-09T18:05:47.516 INFO:teuthology.orchestra.run.smithi078.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.1 [1,091 kB] 2020-11-09T18:05:47.540 INFO:teuthology.orchestra.run.smithi051.stdout:Get:71 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 16.0.0-7048-ga0560c8e-1bionic [331 kB] 2020-11-09T18:05:47.547 INFO:teuthology.orchestra.run.smithi078.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2020-11-09T18:05:47.548 INFO:teuthology.orchestra.run.smithi078.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2020-11-09T18:05:47.548 INFO:teuthology.orchestra.run.smithi078.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2020-11-09T18:05:47.549 INFO:teuthology.orchestra.run.smithi051.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2020-11-09T18:05:47.549 INFO:teuthology.orchestra.run.smithi051.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2020-11-09T18:05:47.614 INFO:teuthology.orchestra.run.smithi051.stdout:Get:74 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 16.0.0-7048-ga0560c8e-1bionic [20.6 kB] 2020-11-09T18:05:47.615 INFO:teuthology.orchestra.run.smithi051.stdout:Get:75 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 16.0.0-7048-ga0560c8e-1bionic [168 kB] 2020-11-09T18:05:47.628 INFO:teuthology.orchestra.run.smithi051.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.1 [1,091 kB] 2020-11-09T18:05:47.649 INFO:teuthology.orchestra.run.smithi051.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2020-11-09T18:05:47.650 INFO:teuthology.orchestra.run.smithi051.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2020-11-09T18:05:47.650 INFO:teuthology.orchestra.run.smithi051.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2020-11-09T18:05:47.654 INFO:teuthology.orchestra.run.smithi078.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2020-11-09T18:05:47.654 INFO:teuthology.orchestra.run.smithi078.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2020-11-09T18:05:47.655 INFO:teuthology.orchestra.run.smithi078.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.3 [330 kB] 2020-11-09T18:05:47.661 INFO:teuthology.orchestra.run.smithi078.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2020-11-09T18:05:47.662 INFO:teuthology.orchestra.run.smithi078.stdout:Get:95 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2020-11-09T18:05:47.662 INFO:teuthology.orchestra.run.smithi078.stdout:Get:96 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2020-11-09T18:05:47.676 INFO:teuthology.orchestra.run.smithi078.stdout:Get:97 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2020-11-09T18:05:47.676 INFO:teuthology.orchestra.run.smithi051.stdout:Get:80 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 16.0.0-7048-ga0560c8e-1bionic [27.7 kB] 2020-11-09T18:05:47.677 INFO:teuthology.orchestra.run.smithi051.stdout:Get:81 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 16.0.0-7048-ga0560c8e-1bionic [330 kB] 2020-11-09T18:05:47.678 INFO:teuthology.orchestra.run.smithi078.stdout:Get:98 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2020-11-09T18:05:47.678 INFO:teuthology.orchestra.run.smithi078.stdout:Get:99 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2020-11-09T18:05:47.784 INFO:teuthology.orchestra.run.smithi051.stdout:Get:82 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 16.0.0-7048-ga0560c8e-1bionic [3,169 kB] 2020-11-09T18:05:47.789 INFO:teuthology.orchestra.run.smithi051.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2020-11-09T18:05:47.789 INFO:teuthology.orchestra.run.smithi051.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2020-11-09T18:05:47.790 INFO:teuthology.orchestra.run.smithi051.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.3 [330 kB] 2020-11-09T18:05:47.796 INFO:teuthology.orchestra.run.smithi051.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2020-11-09T18:05:47.797 INFO:teuthology.orchestra.run.smithi051.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2020-11-09T18:05:47.798 INFO:teuthology.orchestra.run.smithi051.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2020-11-09T18:05:47.798 INFO:teuthology.orchestra.run.smithi051.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2020-11-09T18:05:47.800 INFO:teuthology.orchestra.run.smithi051.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2020-11-09T18:05:47.801 INFO:teuthology.orchestra.run.smithi051.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2020-11-09T18:05:48.337 INFO:teuthology.orchestra.run.smithi078.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 16.0.0-7048-ga0560c8e-1bionic [3,808 B] 2020-11-09T18:05:48.338 INFO:teuthology.orchestra.run.smithi078.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 16.0.0-7048-ga0560c8e-1bionic [562 kB] 2020-11-09T18:05:48.354 INFO:teuthology.orchestra.run.smithi078.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 16.0.0-7048-ga0560c8e-1bionic [1,941 kB] 2020-11-09T18:05:48.424 INFO:teuthology.orchestra.run.smithi078.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 16.0.0-7048-ga0560c8e-1bionic [51.5 kB] 2020-11-09T18:05:48.425 INFO:teuthology.orchestra.run.smithi078.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 16.0.0-7048-ga0560c8e-1bionic [55.9 kB] 2020-11-09T18:05:48.427 INFO:teuthology.orchestra.run.smithi078.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 16.0.0-7048-ga0560c8e-1bionic [2,877 kB] 2020-11-09T18:05:48.538 INFO:teuthology.orchestra.run.smithi078.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 16.0.0-7048-ga0560c8e-1bionic [7,886 kB] 2020-11-09T18:05:48.663 INFO:teuthology.orchestra.run.smithi051.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 16.0.0-7048-ga0560c8e-1bionic [99.1 kB] 2020-11-09T18:05:48.666 INFO:teuthology.orchestra.run.smithi051.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 16.0.0-7048-ga0560c8e-1bionic [414 kB] 2020-11-09T18:05:48.801 INFO:teuthology.orchestra.run.smithi051.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 16.0.0-7048-ga0560c8e-1bionic [20.5 MB] 2020-11-09T18:05:48.830 INFO:teuthology.orchestra.run.smithi078.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 16.0.0-7048-ga0560c8e-1bionic [3,924 B] 2020-11-09T18:05:48.830 INFO:teuthology.orchestra.run.smithi078.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 16.0.0-7048-ga0560c8e-1bionic [22.3 kB] 2020-11-09T18:05:48.830 INFO:teuthology.orchestra.run.smithi078.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 16.0.0-7048-ga0560c8e-1bionic [49.6 MB] 2020-11-09T18:05:50.792 INFO:teuthology.orchestra.run.smithi078.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 16.0.0-7048-ga0560c8e-1bionic [19.5 kB] 2020-11-09T18:05:50.793 INFO:teuthology.orchestra.run.smithi078.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 16.0.0-7048-ga0560c8e-1bionic [9,825 kB] 2020-11-09T18:05:51.189 INFO:teuthology.orchestra.run.smithi078.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 16.0.0-7048-ga0560c8e-1bionic [77.0 kB] 2020-11-09T18:05:51.881 INFO:teuthology.orchestra.run.smithi078.stdout:Fetched 168 MB in 6s (29.5 MB/s) 2020-11-09T18:05:52.038 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-uritemplate. 2020-11-09T18:05:52.076 INFO:teuthology.orchestra.run.smithi078.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102071 files and directories currently installed.) 2020-11-09T18:05:52.078 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2020-11-09T18:05:52.120 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2020-11-09T18:05:52.393 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2020-11-09T18:05:52.405 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../001-liblttng-ust0_2.10.1-1_amd64.deb ... 2020-11-09T18:05:52.465 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2020-11-09T18:05:52.787 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../002-librbd1_16.0.0-7048-ga0560c8e-1bionic_amd64.deb ... 2020-11-09T18:05:52.913 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking librbd1 (16.0.0-7048-ga0560c8e-1bionic) over (12.2.13-0ubuntu0.18.04.4) ... 2020-11-09T18:05:53.466 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../003-librados2_16.0.0-7048-ga0560c8e-1bionic_amd64.deb ... 2020-11-09T18:05:53.562 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking librados2 (16.0.0-7048-ga0560c8e-1bionic) over (12.2.13-0ubuntu0.18.04.4) ... 2020-11-09T18:05:54.186 INFO:teuthology.orchestra.run.smithi051.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 16.0.0-7048-ga0560c8e-1bionic [5,566 kB] 2020-11-09T18:05:54.209 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libcephfs2. 2020-11-09T18:05:54.221 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../004-libcephfs2_16.0.0-7048-ga0560c8e-1bionic_amd64.deb ... 2020-11-09T18:05:54.275 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libcephfs2 (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:05:54.536 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-rados. 2020-11-09T18:05:54.547 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../005-python3-rados_16.0.0-7048-ga0560c8e-1bionic_amd64.deb ... 2020-11-09T18:05:54.578 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-rados (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:05:54.906 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-ceph-argparse. 2020-11-09T18:05:54.918 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../006-python3-ceph-argparse_16.0.0-7048-ga0560c8e-1bionic_all.deb ... 2020-11-09T18:05:54.956 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-ceph-argparse (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:05:55.242 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-cephfs. 2020-11-09T18:05:55.255 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../007-python3-cephfs_16.0.0-7048-ga0560c8e-1bionic_amd64.deb ... 2020-11-09T18:05:55.292 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-cephfs (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:05:55.578 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-ceph-common. 2020-11-09T18:05:55.589 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../008-python3-ceph-common_16.0.0-7048-ga0560c8e-1bionic_all.deb ... 2020-11-09T18:05:55.645 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-ceph-common (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:05:55.897 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-prettytable. 2020-11-09T18:05:55.908 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-3_all.deb ... 2020-11-09T18:05:55.948 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2020-11-09T18:05:56.200 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-rbd. 2020-11-09T18:05:56.212 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../010-python3-rbd_16.0.0-7048-ga0560c8e-1bionic_amd64.deb ... 2020-11-09T18:05:56.252 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-rbd (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:05:56.262 INFO:teuthology.orchestra.run.smithi051.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 16.0.0-7048-ga0560c8e-1bionic [165 kB] 2020-11-09T18:05:56.314 INFO:teuthology.orchestra.run.smithi051.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 16.0.0-7048-ga0560c8e-1bionic [1,361 kB] 2020-11-09T18:05:56.621 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package librabbitmq4:amd64. 2020-11-09T18:05:56.632 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../011-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2020-11-09T18:05:56.672 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-11-09T18:05:56.811 INFO:teuthology.orchestra.run.smithi051.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 16.0.0-7048-ga0560c8e-1bionic [6,652 kB] 2020-11-09T18:05:56.948 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package librdkafka1:amd64. 2020-11-09T18:05:56.960 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../012-librdkafka1_0.11.3-1build1_amd64.deb ... 2020-11-09T18:05:56.990 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2020-11-09T18:05:57.318 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package librgw2. 2020-11-09T18:05:57.330 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../013-librgw2_16.0.0-7048-ga0560c8e-1bionic_amd64.deb ... 2020-11-09T18:05:57.368 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking librgw2 (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:05:57.920 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-rgw. 2020-11-09T18:05:57.933 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../014-python3-rgw_16.0.0-7048-ga0560c8e-1bionic_amd64.deb ... 2020-11-09T18:05:57.970 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-rgw (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:05:58.273 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2020-11-09T18:05:58.286 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../015-libsnappy1v5_1.1.7-1_amd64.deb ... 2020-11-09T18:05:58.315 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2020-11-09T18:05:58.604 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2020-11-09T18:05:58.616 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../016-libleveldb1v5_1.20-2_amd64.deb ... 2020-11-09T18:05:58.654 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2020-11-09T18:05:58.967 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package liboath0. 2020-11-09T18:05:58.977 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../017-liboath0_2.6.1-1_amd64.deb ... 2020-11-09T18:05:59.020 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking liboath0 (2.6.1-1) ... 2020-11-09T18:05:59.150 INFO:teuthology.orchestra.run.smithi051.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 16.0.0-7048-ga0560c8e-1bionic [24.6 MB] 2020-11-09T18:05:59.318 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libradosstriper1. 2020-11-09T18:05:59.331 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../018-libradosstriper1_16.0.0-7048-ga0560c8e-1bionic_amd64.deb ... 2020-11-09T18:05:59.370 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libradosstriper1 (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:05:59.794 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-common. 2020-11-09T18:05:59.806 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../019-ceph-common_16.0.0-7048-ga0560c8e-1bionic_amd64.deb ... 2020-11-09T18:05:59.845 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-common (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:01.744 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-base. 2020-11-09T18:06:01.755 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../020-ceph-base_16.0.0-7048-ga0560c8e-1bionic_amd64.deb ... 2020-11-09T18:06:01.803 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-base (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:02.729 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-dateutil. 2020-11-09T18:06:02.740 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../021-python3-dateutil_2.6.1-1_all.deb ... 2020-11-09T18:06:02.779 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2020-11-09T18:06:03.098 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-asn1crypto. 2020-11-09T18:06:03.112 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../022-python3-asn1crypto_0.24.0-1_all.deb ... 2020-11-09T18:06:03.148 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2020-11-09T18:06:03.397 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-cffi-backend. 2020-11-09T18:06:03.410 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../023-python3-cffi-backend_1.11.5-1_amd64.deb ... 2020-11-09T18:06:03.447 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2020-11-09T18:06:03.720 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-cryptography. 2020-11-09T18:06:03.732 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../024-python3-cryptography_2.1.4-1ubuntu1.4_amd64.deb ... 2020-11-09T18:06:03.769 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.4) ... 2020-11-09T18:06:04.130 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-openssl. 2020-11-09T18:06:04.142 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../025-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2020-11-09T18:06:04.180 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2020-11-09T18:06:04.416 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2020-11-09T18:06:04.428 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../026-ceph-mgr-modules-core_16.0.0-7048-ga0560c8e-1bionic_all.deb ... 2020-11-09T18:06:04.457 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-mgr-modules-core (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:04.812 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-bcrypt. 2020-11-09T18:06:04.825 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../027-python3-bcrypt_3.1.4-2_amd64.deb ... 2020-11-09T18:06:04.862 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2020-11-09T18:06:05.165 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-cherrypy3. 2020-11-09T18:06:05.176 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../028-python3-cherrypy3_8.9.1-2_all.deb ... 2020-11-09T18:06:05.207 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2020-11-09T18:06:05.493 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-jwt. 2020-11-09T18:06:05.506 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../029-python3-jwt_1.5.3+ds1-1_all.deb ... 2020-11-09T18:06:05.535 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2020-11-09T18:06:05.694 INFO:teuthology.orchestra.run.smithi051.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 16.0.0-7048-ga0560c8e-1bionic [3,808 B] 2020-11-09T18:06:05.694 INFO:teuthology.orchestra.run.smithi051.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 16.0.0-7048-ga0560c8e-1bionic [562 kB] 2020-11-09T18:06:05.832 INFO:teuthology.orchestra.run.smithi051.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 16.0.0-7048-ga0560c8e-1bionic [1,941 kB] 2020-11-09T18:06:05.847 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-logutils. 2020-11-09T18:06:05.859 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../030-python3-logutils_0.3.3-5_all.deb ... 2020-11-09T18:06:05.897 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-logutils (0.3.3-5) ... 2020-11-09T18:06:06.183 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-markupsafe. 2020-11-09T18:06:06.189 INFO:teuthology.orchestra.run.smithi051.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 16.0.0-7048-ga0560c8e-1bionic [51.5 kB] 2020-11-09T18:06:06.194 INFO:teuthology.orchestra.run.smithi051.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 16.0.0-7048-ga0560c8e-1bionic [55.9 kB] 2020-11-09T18:06:06.195 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../031-python3-markupsafe_1.0-1build1_amd64.deb ... 2020-11-09T18:06:06.199 INFO:teuthology.orchestra.run.smithi051.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 16.0.0-7048-ga0560c8e-1bionic [2,877 kB] 2020-11-09T18:06:06.234 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2020-11-09T18:06:06.485 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-mako. 2020-11-09T18:06:06.593 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../032-python3-mako_1.0.7+ds1-1_all.deb ... 2020-11-09T18:06:06.593 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2020-11-09T18:06:06.657 INFO:teuthology.orchestra.run.smithi051.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 16.0.0-7048-ga0560c8e-1bionic [7,886 kB] 2020-11-09T18:06:06.852 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-simplegeneric. 2020-11-09T18:06:06.864 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../033-python3-simplegeneric_0.8.1-1_all.deb ... 2020-11-09T18:06:06.915 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2020-11-09T18:06:07.204 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-singledispatch. 2020-11-09T18:06:07.217 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../034-python3-singledispatch_3.4.0.3-2_all.deb ... 2020-11-09T18:06:07.254 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2020-11-09T18:06:07.558 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-webob. 2020-11-09T18:06:07.570 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../035-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2020-11-09T18:06:07.609 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2020-11-09T18:06:07.806 INFO:teuthology.orchestra.run.smithi051.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 16.0.0-7048-ga0560c8e-1bionic [3,924 B] 2020-11-09T18:06:07.807 INFO:teuthology.orchestra.run.smithi051.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 16.0.0-7048-ga0560c8e-1bionic [22.3 kB] 2020-11-09T18:06:07.809 INFO:teuthology.orchestra.run.smithi051.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 16.0.0-7048-ga0560c8e-1bionic [49.6 MB] 2020-11-09T18:06:07.915 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-bs4. 2020-11-09T18:06:07.928 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../036-python3-bs4_4.6.0-1_all.deb ... 2020-11-09T18:06:07.958 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2020-11-09T18:06:08.282 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-waitress. 2020-11-09T18:06:08.294 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../037-python3-waitress_1.0.1-1_all.deb ... 2020-11-09T18:06:08.332 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-waitress (1.0.1-1) ... 2020-11-09T18:06:08.988 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-tempita. 2020-11-09T18:06:08.999 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../038-python3-tempita_0.5.2-2_all.deb ... 2020-11-09T18:06:09.037 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-tempita (0.5.2-2) ... 2020-11-09T18:06:09.353 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-paste. 2020-11-09T18:06:09.364 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../039-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2020-11-09T18:06:09.403 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-11-09T18:06:09.695 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2020-11-09T18:06:09.706 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../040-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2020-11-09T18:06:09.745 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2020-11-09T18:06:10.019 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pastedeploy. 2020-11-09T18:06:10.032 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../041-python3-pastedeploy_1.5.2-4_all.deb ... 2020-11-09T18:06:10.078 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2020-11-09T18:06:10.375 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-webtest. 2020-11-09T18:06:10.387 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../042-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2020-11-09T18:06:10.425 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2020-11-09T18:06:10.761 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pecan. 2020-11-09T18:06:10.774 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../043-python3-pecan_1.2.1-2_all.deb ... 2020-11-09T18:06:10.812 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pecan (1.2.1-2) ... 2020-11-09T18:06:11.167 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-werkzeug. 2020-11-09T18:06:11.179 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../044-python3-werkzeug_0.14.1+dfsg1-1_all.deb ... 2020-11-09T18:06:11.210 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1) ... 2020-11-09T18:06:11.570 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-mgr. 2020-11-09T18:06:11.582 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../045-ceph-mgr_16.0.0-7048-ga0560c8e-1bionic_amd64.deb ... 2020-11-09T18:06:11.621 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-mgr (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:12.098 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-mon. 2020-11-09T18:06:12.110 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../046-ceph-mon_16.0.0-7048-ga0560c8e-1bionic_amd64.deb ... 2020-11-09T18:06:12.149 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-mon (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:13.083 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-osd. 2020-11-09T18:06:13.095 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../047-ceph-osd_16.0.0-7048-ga0560c8e-1bionic_amd64.deb ... 2020-11-09T18:06:13.135 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-osd (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:15.081 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph. 2020-11-09T18:06:15.093 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../048-ceph_16.0.0-7048-ga0560c8e-1bionic_amd64.deb ... 2020-11-09T18:06:15.098 INFO:teuthology.orchestra.run.smithi051.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 16.0.0-7048-ga0560c8e-1bionic [19.5 kB] 2020-11-09T18:06:15.099 INFO:teuthology.orchestra.run.smithi051.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 16.0.0-7048-ga0560c8e-1bionic [9,825 kB] 2020-11-09T18:06:15.131 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:15.450 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-fuse. 2020-11-09T18:06:15.462 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../049-ceph-fuse_16.0.0-7048-ga0560c8e-1bionic_amd64.deb ... 2020-11-09T18:06:15.500 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-fuse (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:15.915 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-mds. 2020-11-09T18:06:15.928 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../050-ceph-mds_16.0.0-7048-ga0560c8e-1bionic_amd64.deb ... 2020-11-09T18:06:15.966 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-mds (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:16.434 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package cephadm. 2020-11-09T18:06:16.446 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../051-cephadm_16.0.0-7048-ga0560c8e-1bionic_amd64.deb ... 2020-11-09T18:06:16.484 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking cephadm (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:16.794 INFO:teuthology.orchestra.run.smithi051.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-dang-webdir-empty-object/a0560c8e5c6c33c4655e9d2a8b022793ae57419f/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 16.0.0-7048-ga0560c8e-1bionic [77.0 kB] 2020-11-09T18:06:16.807 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-jinja2. 2020-11-09T18:06:16.820 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../052-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2020-11-09T18:06:16.857 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-11-09T18:06:17.153 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2020-11-09T18:06:17.166 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../053-ceph-mgr-cephadm_16.0.0-7048-ga0560c8e-1bionic_all.deb ... 2020-11-09T18:06:17.204 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-mgr-cephadm (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:17.521 INFO:teuthology.orchestra.run.smithi051.stdout:Fetched 168 MB in 31s (5,392 kB/s) 2020-11-09T18:06:17.536 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-repoze.lru. 2020-11-09T18:06:17.548 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../054-python3-repoze.lru_0.7-1_all.deb ... 2020-11-09T18:06:17.586 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2020-11-09T18:06:17.666 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-uritemplate. 2020-11-09T18:06:17.703 INFO:teuthology.orchestra.run.smithi051.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102071 files and directories currently installed.) 2020-11-09T18:06:17.706 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2020-11-09T18:06:17.748 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2020-11-09T18:06:17.889 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-routes. 2020-11-09T18:06:17.901 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../055-python3-routes_2.4.1-1_all.deb ... 2020-11-09T18:06:17.939 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-routes (2.4.1-1) ... 2020-11-09T18:06:18.014 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2020-11-09T18:06:18.025 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../001-liblttng-ust0_2.10.1-1_amd64.deb ... 2020-11-09T18:06:18.107 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2020-11-09T18:06:18.225 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2020-11-09T18:06:18.238 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../056-ceph-mgr-dashboard_16.0.0-7048-ga0560c8e-1bionic_all.deb ... 2020-11-09T18:06:18.275 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-mgr-dashboard (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:18.454 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../002-librbd1_16.0.0-7048-ga0560c8e-1bionic_amd64.deb ... 2020-11-09T18:06:18.579 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking librbd1 (16.0.0-7048-ga0560c8e-1bionic) over (12.2.13-0ubuntu0.18.04.4) ... 2020-11-09T18:06:19.805 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../003-librados2_16.0.0-7048-ga0560c8e-1bionic_amd64.deb ... 2020-11-09T18:06:19.928 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking librados2 (16.0.0-7048-ga0560c8e-1bionic) over (12.2.13-0ubuntu0.18.04.4) ... 2020-11-09T18:06:19.988 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-numpy. 2020-11-09T18:06:20.002 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../057-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2020-11-09T18:06:20.048 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2020-11-09T18:06:20.647 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package libcephfs2. 2020-11-09T18:06:20.659 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../004-libcephfs2_16.0.0-7048-ga0560c8e-1bionic_amd64.deb ... 2020-11-09T18:06:20.684 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-decorator. 2020-11-09T18:06:20.689 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking libcephfs2 (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:20.697 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../058-python3-decorator_4.1.2-1_all.deb ... 2020-11-09T18:06:20.741 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-decorator (4.1.2-1) ... 2020-11-09T18:06:20.974 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-rados. 2020-11-09T18:06:20.987 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../005-python3-rados_16.0.0-7048-ga0560c8e-1bionic_amd64.deb ... 2020-11-09T18:06:21.025 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-rados (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:21.053 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-scipy. 2020-11-09T18:06:21.067 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../059-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2020-11-09T18:06:21.103 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2020-11-09T18:06:21.361 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-ceph-argparse. 2020-11-09T18:06:21.373 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../006-python3-ceph-argparse_16.0.0-7048-ga0560c8e-1bionic_all.deb ... 2020-11-09T18:06:21.413 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-ceph-argparse (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:21.648 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-cephfs. 2020-11-09T18:06:21.661 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../007-python3-cephfs_16.0.0-7048-ga0560c8e-1bionic_amd64.deb ... 2020-11-09T18:06:21.698 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-cephfs (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:22.026 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-ceph-common. 2020-11-09T18:06:22.037 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../008-python3-ceph-common_16.0.0-7048-ga0560c8e-1bionic_all.deb ... 2020-11-09T18:06:22.076 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-ceph-common (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:22.263 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-sklearn-lib. 2020-11-09T18:06:22.276 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../060-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2020-11-09T18:06:22.321 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2020-11-09T18:06:22.387 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-prettytable. 2020-11-09T18:06:22.399 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-3_all.deb ... 2020-11-09T18:06:22.437 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2020-11-09T18:06:22.676 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-rbd. 2020-11-09T18:06:22.687 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../010-python3-rbd_16.0.0-7048-ga0560c8e-1bionic_amd64.deb ... 2020-11-09T18:06:22.726 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-rbd (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:22.754 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-joblib. 2020-11-09T18:06:22.768 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../061-python3-joblib_0.11-1_all.deb ... 2020-11-09T18:06:22.804 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-joblib (0.11-1) ... 2020-11-09T18:06:23.129 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package librabbitmq4:amd64. 2020-11-09T18:06:23.140 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../011-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2020-11-09T18:06:23.187 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-11-09T18:06:23.204 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-sklearn. 2020-11-09T18:06:23.217 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../062-python3-sklearn_0.19.1-3_all.deb ... 2020-11-09T18:06:23.251 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2020-11-09T18:06:23.515 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package librdkafka1:amd64. 2020-11-09T18:06:23.526 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../012-librdkafka1_0.11.3-1build1_amd64.deb ... 2020-11-09T18:06:23.565 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2020-11-09T18:06:23.833 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2020-11-09T18:06:23.846 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../063-ceph-mgr-diskprediction-local_16.0.0-7048-ga0560c8e-1bionic_all.deb ... 2020-11-09T18:06:23.851 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package librgw2. 2020-11-09T18:06:23.864 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../013-librgw2_16.0.0-7048-ga0560c8e-1bionic_amd64.deb ... 2020-11-09T18:06:23.893 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-mgr-diskprediction-local (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:23.893 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking librgw2 (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:24.445 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-rgw. 2020-11-09T18:06:24.455 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../014-python3-rgw_16.0.0-7048-ga0560c8e-1bionic_amd64.deb ... 2020-11-09T18:06:24.495 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-rgw (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:24.789 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2020-11-09T18:06:24.802 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../015-libsnappy1v5_1.1.7-1_amd64.deb ... 2020-11-09T18:06:24.839 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2020-11-09T18:06:24.998 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-monotonic. 2020-11-09T18:06:25.012 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../064-python3-monotonic_1.1-2_all.deb ... 2020-11-09T18:06:25.057 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-monotonic (1.1-2) ... 2020-11-09T18:06:25.157 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2020-11-09T18:06:25.170 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../016-libleveldb1v5_1.20-2_amd64.deb ... 2020-11-09T18:06:25.211 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2020-11-09T18:06:25.401 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-fasteners. 2020-11-09T18:06:25.414 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../065-python3-fasteners_0.12.0-3_all.deb ... 2020-11-09T18:06:25.452 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2020-11-09T18:06:25.577 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package liboath0. 2020-11-09T18:06:25.587 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../017-liboath0_2.6.1-1_amd64.deb ... 2020-11-09T18:06:25.619 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking liboath0 (2.6.1-1) ... 2020-11-09T18:06:25.752 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-httplib2. 2020-11-09T18:06:25.764 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../066-python3-httplib2_0.9.2+dfsg-1ubuntu0.2_all.deb ... 2020-11-09T18:06:25.803 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-11-09T18:06:25.911 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package libradosstriper1. 2020-11-09T18:06:25.925 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../018-libradosstriper1_16.0.0-7048-ga0560c8e-1bionic_amd64.deb ... 2020-11-09T18:06:25.963 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking libradosstriper1 (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:26.111 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pyasn1. 2020-11-09T18:06:26.123 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../067-python3-pyasn1_0.4.2-3_all.deb ... 2020-11-09T18:06:26.161 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2020-11-09T18:06:26.387 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package ceph-common. 2020-11-09T18:06:26.398 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../019-ceph-common_16.0.0-7048-ga0560c8e-1bionic_amd64.deb ... 2020-11-09T18:06:26.437 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking ceph-common (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:26.472 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pyasn1-modules. 2020-11-09T18:06:26.485 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../068-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2020-11-09T18:06:26.523 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2020-11-09T18:06:26.858 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-rsa. 2020-11-09T18:06:26.871 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../069-python3-rsa_3.4.2-1_all.deb ... 2020-11-09T18:06:26.909 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-rsa (3.4.2-1) ... 2020-11-09T18:06:27.218 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-sqlalchemy. 2020-11-09T18:06:27.231 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../070-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2020-11-09T18:06:27.277 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-11-09T18:06:27.721 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-oauth2client. 2020-11-09T18:06:27.732 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../071-python3-oauth2client_4.1.2-3_all.deb ... 2020-11-09T18:06:27.771 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2020-11-09T18:06:28.096 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-setuptools. 2020-11-09T18:06:28.107 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../072-python3-setuptools_39.0.1-2_all.deb ... 2020-11-09T18:06:28.146 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2020-11-09T18:06:28.354 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package ceph-base. 2020-11-09T18:06:28.365 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../020-ceph-base_16.0.0-7048-ga0560c8e-1bionic_amd64.deb ... 2020-11-09T18:06:28.406 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking ceph-base (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:28.501 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-websocket. 2020-11-09T18:06:28.512 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../073-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2020-11-09T18:06:28.550 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2020-11-09T18:06:28.850 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-kubernetes. 2020-11-09T18:06:28.864 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../074-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2020-11-09T18:06:28.902 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2020-11-09T18:06:29.305 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-dateutil. 2020-11-09T18:06:29.310 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2020-11-09T18:06:29.316 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../021-python3-dateutil_2.6.1-1_all.deb ... 2020-11-09T18:06:29.323 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../075-ceph-mgr-k8sevents_16.0.0-7048-ga0560c8e-1bionic_all.deb ... 2020-11-09T18:06:29.355 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2020-11-09T18:06:29.362 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-mgr-k8sevents (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:29.650 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-asn1crypto. 2020-11-09T18:06:29.661 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../022-python3-asn1crypto_0.24.0-1_all.deb ... 2020-11-09T18:06:29.678 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-json-pointer. 2020-11-09T18:06:29.691 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../076-python3-json-pointer_1.10-1_all.deb ... 2020-11-09T18:06:29.692 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2020-11-09T18:06:29.733 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-json-pointer (1.10-1) ... 2020-11-09T18:06:29.972 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-cffi-backend. 2020-11-09T18:06:29.983 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../023-python3-cffi-backend_1.11.5-1_amd64.deb ... 2020-11-09T18:06:30.023 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2020-11-09T18:06:30.088 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-jsonpatch. 2020-11-09T18:06:30.101 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../077-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2020-11-09T18:06:30.150 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-11-09T18:06:30.339 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-cryptography. 2020-11-09T18:06:30.350 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../024-python3-cryptography_2.1.4-1ubuntu1.4_amd64.deb ... 2020-11-09T18:06:30.389 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.4) ... 2020-11-09T18:06:30.482 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-mgr-rook. 2020-11-09T18:06:30.493 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../078-ceph-mgr-rook_16.0.0-7048-ga0560c8e-1bionic_all.deb ... 2020-11-09T18:06:30.531 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-mgr-rook (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:30.751 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-openssl. 2020-11-09T18:06:30.763 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../025-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2020-11-09T18:06:30.801 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2020-11-09T18:06:30.843 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libonig4:amd64. 2020-11-09T18:06:30.856 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../079-libonig4_6.7.0-1_amd64.deb ... 2020-11-09T18:06:30.901 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2020-11-09T18:06:31.070 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2020-11-09T18:06:31.083 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../026-ceph-mgr-modules-core_16.0.0-7048-ga0560c8e-1bionic_all.deb ... 2020-11-09T18:06:31.120 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking ceph-mgr-modules-core (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:31.212 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libjq1:amd64. 2020-11-09T18:06:31.226 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../080-libjq1_1.5+dfsg-2_amd64.deb ... 2020-11-09T18:06:31.347 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2020-11-09T18:06:31.475 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-bcrypt. 2020-11-09T18:06:31.486 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../027-python3-bcrypt_3.1.4-2_amd64.deb ... 2020-11-09T18:06:31.525 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2020-11-09T18:06:31.600 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package jq. 2020-11-09T18:06:31.611 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../081-jq_1.5+dfsg-2_amd64.deb ... 2020-11-09T18:06:31.660 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking jq (1.5+dfsg-2) ... 2020-11-09T18:06:31.827 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-cherrypy3. 2020-11-09T18:06:31.839 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../028-python3-cherrypy3_8.9.1-2_all.deb ... 2020-11-09T18:06:31.878 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2020-11-09T18:06:31.896 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package socat. 2020-11-09T18:06:31.908 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../082-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2020-11-09T18:06:31.953 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2020-11-09T18:06:32.272 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-jwt. 2020-11-09T18:06:32.281 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libxslt1.1:amd64. 2020-11-09T18:06:32.284 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../029-python3-jwt_1.5.3+ds1-1_all.deb ... 2020-11-09T18:06:32.294 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../083-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2020-11-09T18:06:32.322 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2020-11-09T18:06:32.332 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-11-09T18:06:32.600 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-logutils. 2020-11-09T18:06:32.612 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../030-python3-logutils_0.3.3-5_all.deb ... 2020-11-09T18:06:32.615 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package xmlstarlet. 2020-11-09T18:06:32.627 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../084-xmlstarlet_1.6.1-2_amd64.deb ... 2020-11-09T18:06:32.642 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-logutils (0.3.3-5) ... 2020-11-09T18:06:32.667 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2020-11-09T18:06:32.928 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-markupsafe. 2020-11-09T18:06:32.939 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../031-python3-markupsafe_1.0-1build1_amd64.deb ... 2020-11-09T18:06:32.978 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2020-11-09T18:06:33.018 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-test. 2020-11-09T18:06:33.030 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../085-ceph-test_16.0.0-7048-ga0560c8e-1bionic_amd64.deb ... 2020-11-09T18:06:33.067 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-test (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:33.272 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-mako. 2020-11-09T18:06:33.284 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../032-python3-mako_1.0.7+ds1-1_all.deb ... 2020-11-09T18:06:33.322 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2020-11-09T18:06:33.738 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-simplegeneric. 2020-11-09T18:06:33.752 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../033-python3-simplegeneric_0.8.1-1_all.deb ... 2020-11-09T18:06:33.791 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2020-11-09T18:06:34.060 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-singledispatch. 2020-11-09T18:06:34.074 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../034-python3-singledispatch_3.4.0.3-2_all.deb ... 2020-11-09T18:06:34.110 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2020-11-09T18:06:34.424 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-webob. 2020-11-09T18:06:34.437 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../035-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2020-11-09T18:06:34.474 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2020-11-09T18:06:34.793 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-bs4. 2020-11-09T18:06:34.805 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../036-python3-bs4_4.6.0-1_all.deb ... 2020-11-09T18:06:34.843 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2020-11-09T18:06:35.181 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-waitress. 2020-11-09T18:06:35.188 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../037-python3-waitress_1.0.1-1_all.deb ... 2020-11-09T18:06:35.232 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-waitress (1.0.1-1) ... 2020-11-09T18:06:35.572 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-tempita. 2020-11-09T18:06:35.586 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../038-python3-tempita_0.5.2-2_all.deb ... 2020-11-09T18:06:35.621 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-tempita (0.5.2-2) ... 2020-11-09T18:06:35.935 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-paste. 2020-11-09T18:06:35.948 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../039-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2020-11-09T18:06:35.977 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-11-09T18:06:36.303 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2020-11-09T18:06:36.315 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../040-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2020-11-09T18:06:36.360 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2020-11-09T18:06:36.695 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-pastedeploy. 2020-11-09T18:06:36.706 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../041-python3-pastedeploy_1.5.2-4_all.deb ... 2020-11-09T18:06:36.745 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2020-11-09T18:06:36.753 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libcephfs-dev. 2020-11-09T18:06:36.764 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../086-libcephfs-dev_16.0.0-7048-ga0560c8e-1bionic_amd64.deb ... 2020-11-09T18:06:36.804 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libcephfs-dev (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:37.026 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-webtest. 2020-11-09T18:06:37.039 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../042-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2020-11-09T18:06:37.075 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package nvme-cli. 2020-11-09T18:06:37.076 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2020-11-09T18:06:37.088 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../087-nvme-cli_1.5-1ubuntu1_amd64.deb ... 2020-11-09T18:06:37.126 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking nvme-cli (1.5-1ubuntu1) ... 2020-11-09T18:06:37.405 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-pecan. 2020-11-09T18:06:37.440 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../043-python3-pecan_1.2.1-2_all.deb ... 2020-11-09T18:06:37.455 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-pecan (1.2.1-2) ... 2020-11-09T18:06:37.456 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python-attr. 2020-11-09T18:06:37.469 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../088-python-attr_17.4.0-2_all.deb ... 2020-11-09T18:06:37.514 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python-attr (17.4.0-2) ... 2020-11-09T18:06:37.796 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-werkzeug. 2020-11-09T18:06:37.809 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../044-python3-werkzeug_0.14.1+dfsg1-1_all.deb ... 2020-11-09T18:06:37.840 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python-funcsigs. 2020-11-09T18:06:37.840 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1) ... 2020-11-09T18:06:37.851 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../089-python-funcsigs_1.0.2-4_all.deb ... 2020-11-09T18:06:37.896 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2020-11-09T18:06:38.177 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package ceph-mgr. 2020-11-09T18:06:38.190 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../045-ceph-mgr_16.0.0-7048-ga0560c8e-1bionic_amd64.deb ... 2020-11-09T18:06:38.217 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python-pluggy. 2020-11-09T18:06:38.227 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking ceph-mgr (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:38.231 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../090-python-pluggy_0.6.0-1_all.deb ... 2020-11-09T18:06:38.267 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python-pluggy (0.6.0-1) ... 2020-11-09T18:06:38.574 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python-py. 2020-11-09T18:06:38.587 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../091-python-py_1.5.2-1_all.deb ... 2020-11-09T18:06:38.630 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python-py (1.5.2-1) ... 2020-11-09T18:06:38.713 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package ceph-mon. 2020-11-09T18:06:38.724 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../046-ceph-mon_16.0.0-7048-ga0560c8e-1bionic_amd64.deb ... 2020-11-09T18:06:38.763 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking ceph-mon (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:38.972 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python-pytest. 2020-11-09T18:06:38.985 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../092-python-pytest_3.3.2-2_all.deb ... 2020-11-09T18:06:39.022 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python-pytest (3.3.2-2) ... 2020-11-09T18:06:39.609 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-attr. 2020-11-09T18:06:39.623 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../093-python3-attr_17.4.0-2_all.deb ... 2020-11-09T18:06:39.659 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-attr (17.4.0-2) ... 2020-11-09T18:06:39.700 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package ceph-osd. 2020-11-09T18:06:39.711 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../047-ceph-osd_16.0.0-7048-ga0560c8e-1bionic_amd64.deb ... 2020-11-09T18:06:39.755 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking ceph-osd (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:39.976 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-cycler. 2020-11-09T18:06:39.987 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../094-python3-cycler_0.10.0-1_all.deb ... 2020-11-09T18:06:40.040 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-cycler (0.10.0-1) ... 2020-11-09T18:06:40.329 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-webencodings. 2020-11-09T18:06:40.342 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../095-python3-webencodings_0.5-2_all.deb ... 2020-11-09T18:06:40.381 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-webencodings (0.5-2) ... 2020-11-09T18:06:40.682 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-html5lib. 2020-11-09T18:06:40.695 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../096-python3-html5lib_0.999999999-1_all.deb ... 2020-11-09T18:06:40.740 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2020-11-09T18:06:41.036 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-lxml:amd64. 2020-11-09T18:06:41.049 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../097-python3-lxml_4.2.1-1ubuntu0.1_amd64.deb ... 2020-11-09T18:06:41.086 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-11-09T18:06:41.556 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pyparsing. 2020-11-09T18:06:41.568 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../098-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2020-11-09T18:06:41.605 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2020-11-09T18:06:41.785 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package ceph. 2020-11-09T18:06:41.796 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../048-ceph_16.0.0-7048-ga0560c8e-1bionic_amd64.deb ... 2020-11-09T18:06:41.835 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking ceph (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:41.925 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-tz. 2020-11-09T18:06:41.939 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../099-python3-tz_2018.3-2_all.deb ... 2020-11-09T18:06:41.983 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-tz (2018.3-2) ... 2020-11-09T18:06:42.275 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-matplotlib. 2020-11-09T18:06:42.287 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../100-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2020-11-09T18:06:42.333 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2020-11-09T18:06:42.381 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package ceph-fuse. 2020-11-09T18:06:42.394 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../049-ceph-fuse_16.0.0-7048-ga0560c8e-1bionic_amd64.deb ... 2020-11-09T18:06:42.439 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking ceph-fuse (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:42.830 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package ceph-mds. 2020-11-09T18:06:42.844 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../050-ceph-mds_16.0.0-7048-ga0560c8e-1bionic_amd64.deb ... 2020-11-09T18:06:42.881 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking ceph-mds (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:42.973 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-olefile. 2020-11-09T18:06:42.985 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../101-python3-olefile_0.45.1-1_all.deb ... 2020-11-09T18:06:43.022 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-olefile (0.45.1-1) ... 2020-11-09T18:06:43.333 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package cephadm. 2020-11-09T18:06:43.342 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pastescript. 2020-11-09T18:06:43.345 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../051-cephadm_16.0.0-7048-ga0560c8e-1bionic_amd64.deb ... 2020-11-09T18:06:43.355 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../102-python3-pastescript_2.0.2-2_all.deb ... 2020-11-09T18:06:43.386 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking cephadm (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:43.392 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2020-11-09T18:06:43.707 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pil:amd64. 2020-11-09T18:06:43.722 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../103-python3-pil_5.1.0-1ubuntu0.3_amd64.deb ... 2020-11-09T18:06:43.724 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-jinja2. 2020-11-09T18:06:43.735 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../052-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2020-11-09T18:06:43.765 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-11-09T18:06:43.776 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-11-09T18:06:44.108 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2020-11-09T18:06:44.120 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../053-ceph-mgr-cephadm_16.0.0-7048-ga0560c8e-1bionic_all.deb ... 2020-11-09T18:06:44.136 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pluggy. 2020-11-09T18:06:44.147 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../104-python3-pluggy_0.6.0-1_all.deb ... 2020-11-09T18:06:44.159 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking ceph-mgr-cephadm (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:44.185 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2020-11-09T18:06:44.488 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-py. 2020-11-09T18:06:44.502 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../105-python3-py_1.5.2-1_all.deb ... 2020-11-09T18:06:44.507 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-repoze.lru. 2020-11-09T18:06:44.520 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../054-python3-repoze.lru_0.7-1_all.deb ... 2020-11-09T18:06:44.538 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-py (1.5.2-1) ... 2020-11-09T18:06:44.557 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2020-11-09T18:06:44.871 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pyinotify. 2020-11-09T18:06:44.884 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../106-python3-pyinotify_0.9.6-1_all.deb ... 2020-11-09T18:06:44.929 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2020-11-09T18:06:45.060 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-routes. 2020-11-09T18:06:45.071 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../055-python3-routes_2.4.1-1_all.deb ... 2020-11-09T18:06:45.110 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-routes (2.4.1-1) ... 2020-11-09T18:06:45.232 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pytest. 2020-11-09T18:06:45.246 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../107-python3-pytest_3.3.2-2_all.deb ... 2020-11-09T18:06:45.291 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pytest (3.3.2-2) ... 2020-11-09T18:06:45.404 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2020-11-09T18:06:45.417 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../056-ceph-mgr-dashboard_16.0.0-7048-ga0560c8e-1bionic_all.deb ... 2020-11-09T18:06:45.454 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking ceph-mgr-dashboard (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:45.602 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-simplejson. 2020-11-09T18:06:45.615 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../108-python3-simplejson_3.13.2-1_amd64.deb ... 2020-11-09T18:06:45.644 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2020-11-09T18:06:45.939 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-tk:amd64. 2020-11-09T18:06:45.953 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../109-python3-tk_3.6.9-1~18.04_amd64.deb ... 2020-11-09T18:06:45.992 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2020-11-09T18:06:46.344 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-numpy. 2020-11-09T18:06:46.357 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../057-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2020-11-09T18:06:46.372 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package radosgw. 2020-11-09T18:06:46.385 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../110-radosgw_16.0.0-7048-ga0560c8e-1bionic_amd64.deb ... 2020-11-09T18:06:46.395 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2020-11-09T18:06:46.413 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking radosgw (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:46.989 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-decorator. 2020-11-09T18:06:47.000 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../058-python3-decorator_4.1.2-1_all.deb ... 2020-11-09T18:06:47.030 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-decorator (4.1.2-1) ... 2020-11-09T18:06:47.326 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-scipy. 2020-11-09T18:06:47.338 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../059-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2020-11-09T18:06:47.377 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2020-11-09T18:06:47.425 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package rbd-fuse. 2020-11-09T18:06:47.438 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../111-rbd-fuse_16.0.0-7048-ga0560c8e-1bionic_amd64.deb ... 2020-11-09T18:06:47.475 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking rbd-fuse (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:47.789 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-logutils (0.3.3-5) ... 2020-11-09T18:06:48.084 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1) ... 2020-11-09T18:06:48.453 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2020-11-09T18:06:48.560 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-sklearn-lib. 2020-11-09T18:06:48.562 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-11-09T18:06:48.572 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../060-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2020-11-09T18:06:48.610 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2020-11-09T18:06:49.062 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-joblib. 2020-11-09T18:06:49.073 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../061-python3-joblib_0.11-1_all.deb ... 2020-11-09T18:06:49.113 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-joblib (0.11-1) ... 2020-11-09T18:06:49.148 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-11-09T18:06:49.441 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-sklearn. 2020-11-09T18:06:49.453 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../062-python3-sklearn_0.19.1-3_all.deb ... 2020-11-09T18:06:49.486 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2020-11-09T18:06:49.525 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2020-11-09T18:06:49.817 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-webencodings (0.5-2) ... 2020-11-09T18:06:50.117 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python-attr (17.4.0-2) ... 2020-11-09T18:06:50.124 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2020-11-09T18:06:50.137 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../063-ceph-mgr-diskprediction-local_16.0.0-7048-ga0560c8e-1bionic_all.deb ... 2020-11-09T18:06:50.182 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking ceph-mgr-diskprediction-local (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:50.324 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2020-11-09T18:06:50.451 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-py (1.5.2-1) ... 2020-11-09T18:06:50.785 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-waitress (1.0.1-1) ... 2020-11-09T18:06:51.152 INFO:teuthology.orchestra.run.smithi078.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2020-11-09T18:06:51.256 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-monotonic. 2020-11-09T18:06:51.270 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../064-python3-monotonic_1.1-2_all.deb ... 2020-11-09T18:06:51.316 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-monotonic (1.1-2) ... 2020-11-09T18:06:51.447 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-olefile (0.45.1-1) ... 2020-11-09T18:06:51.601 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-fasteners. 2020-11-09T18:06:51.615 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../065-python3-fasteners_0.12.0-3_all.deb ... 2020-11-09T18:06:51.644 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2020-11-09T18:06:51.740 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-ceph-argparse (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:51.968 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-httplib2. 2020-11-09T18:06:51.982 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../066-python3-httplib2_0.9.2+dfsg-1ubuntu0.2_all.deb ... 2020-11-09T18:06:52.018 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-11-09T18:06:52.047 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-tempita (0.5.2-2) ... 2020-11-09T18:06:52.310 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-pyasn1. 2020-11-09T18:06:52.323 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../067-python3-pyasn1_0.4.2-3_all.deb ... 2020-11-09T18:06:52.357 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2020-11-09T18:06:52.360 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2020-11-09T18:06:52.483 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2020-11-09T18:06:52.680 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-pyasn1-modules. 2020-11-09T18:06:52.692 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../068-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2020-11-09T18:06:52.722 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2020-11-09T18:06:53.069 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-rsa. 2020-11-09T18:06:54.246 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2020-11-09T18:06:54.246 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2020-11-09T18:06:54.247 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python-py (1.5.2-1) ... 2020-11-09T18:06:54.247 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-simplejson (3.13.2-1) ... 2020-11-09T18:06:54.247 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../069-python3-rsa_3.4.2-1_all.deb ... 2020-11-09T18:06:54.247 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-rsa (3.4.2-1) ... 2020-11-09T18:06:54.247 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-sqlalchemy. 2020-11-09T18:06:54.247 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../070-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2020-11-09T18:06:54.248 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-11-09T18:06:54.248 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-oauth2client. 2020-11-09T18:06:54.248 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../071-python3-oauth2client_4.1.2-3_all.deb ... 2020-11-09T18:06:54.248 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2020-11-09T18:06:54.383 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-setuptools. 2020-11-09T18:06:54.396 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../072-python3-setuptools_39.0.1-2_all.deb ... 2020-11-09T18:06:54.433 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2020-11-09T18:06:54.516 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-json-pointer (1.10-1) ... 2020-11-09T18:06:54.604 INFO:teuthology.orchestra.run.smithi078.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2020-11-09T18:06:54.813 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-websocket. 2020-11-09T18:06:54.826 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../073-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2020-11-09T18:06:54.841 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2020-11-09T18:06:54.864 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2020-11-09T18:06:54.967 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python-pluggy (0.6.0-1) ... 2020-11-09T18:06:55.155 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-kubernetes. 2020-11-09T18:06:55.168 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../074-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2020-11-09T18:06:55.169 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2020-11-09T18:06:55.205 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2020-11-09T18:06:55.487 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2020-11-09T18:06:55.607 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2020-11-09T18:06:55.622 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../075-ceph-mgr-k8sevents_16.0.0-7048-ga0560c8e-1bionic_all.deb ... 2020-11-09T18:06:55.666 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking ceph-mgr-k8sevents (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:55.796 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-cycler (0.10.0-1) ... 2020-11-09T18:06:55.978 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-json-pointer. 2020-11-09T18:06:55.991 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../076-python3-json-pointer_1.10-1_all.deb ... 2020-11-09T18:06:56.041 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-json-pointer (1.10-1) ... 2020-11-09T18:06:56.081 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2020-11-09T18:06:56.386 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-jsonpatch. 2020-11-09T18:06:56.398 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../077-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2020-11-09T18:06:56.440 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-11-09T18:06:56.457 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-bs4 (4.6.0-1) ... 2020-11-09T18:06:56.728 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package ceph-mgr-rook. 2020-11-09T18:06:56.742 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../078-ceph-mgr-rook_16.0.0-7048-ga0560c8e-1bionic_all.deb ... 2020-11-09T18:06:56.783 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pluggy (0.6.0-1) ... 2020-11-09T18:06:56.786 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking ceph-mgr-rook (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:57.071 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up cephadm (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:57.115 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package libonig4:amd64. 2020-11-09T18:06:57.126 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../079-libonig4_6.7.0-1_amd64.deb ... 2020-11-09T18:06:57.164 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2020-11-09T18:06:57.526 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package libjq1:amd64. 2020-11-09T18:06:57.538 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../080-libjq1_1.5+dfsg-2_amd64.deb ... 2020-11-09T18:06:57.653 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2020-11-09T18:06:57.697 INFO:teuthology.orchestra.run.smithi078.stdout:Adding system user cephadm....done 2020-11-09T18:06:57.861 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2020-11-09T18:06:57.914 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package jq. 2020-11-09T18:06:57.926 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../081-jq_1.5+dfsg-2_amd64.deb ... 2020-11-09T18:06:57.964 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking jq (1.5+dfsg-2) ... 2020-11-09T18:06:58.163 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-11-09T18:06:58.201 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package socat. 2020-11-09T18:06:58.213 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../082-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2020-11-09T18:06:58.242 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2020-11-09T18:06:58.264 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up nvme-cli (1.5-1ubuntu1) ... 2020-11-09T18:06:58.373 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2020-11-09T18:06:58.585 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package libxslt1.1:amd64. 2020-11-09T18:06:58.596 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../083-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2020-11-09T18:06:58.647 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-11-09T18:06:58.728 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-prettytable (0.7.2-3) ... 2020-11-09T18:06:58.930 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package xmlstarlet. 2020-11-09T18:06:58.943 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../084-xmlstarlet_1.6.1-2_amd64.deb ... 2020-11-09T18:06:58.980 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2020-11-09T18:06:59.048 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2020-11-09T18:06:59.164 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2020-11-09T18:06:59.331 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package ceph-test. 2020-11-09T18:06:59.344 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../085-ceph-test_16.0.0-7048-ga0560c8e-1bionic_amd64.deb ... 2020-11-09T18:06:59.382 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking ceph-test (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:59.474 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-ceph-common (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:06:59.901 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-monotonic (1.1-2) ... 2020-11-09T18:07:00.195 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2020-11-09T18:07:00.319 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-11-09T18:07:00.662 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-html5lib (0.999999999-1) ... 2020-11-09T18:07:01.010 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-11-09T18:07:01.099 INFO:teuthology.orchestra.run.smithi078.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2020-11-09T18:07:01.140 INFO:teuthology.orchestra.run.smithi078.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2020-11-09T18:07:01.378 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-repoze.lru (0.7-1) ... 2020-11-09T18:07:01.679 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-11-09T18:07:01.983 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2020-11-09T18:07:02.307 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-attr (17.4.0-2) ... 2020-11-09T18:07:02.626 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python-funcsigs (1.0.2-4) ... 2020-11-09T18:07:02.826 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-11-09T18:07:03.061 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package libcephfs-dev. 2020-11-09T18:07:03.073 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../086-libcephfs-dev_16.0.0-7048-ga0560c8e-1bionic_amd64.deb ... 2020-11-09T18:07:03.112 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking libcephfs-dev (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:03.233 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2020-11-09T18:07:03.323 INFO:teuthology.orchestra.run.smithi078.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2020-11-09T18:07:03.378 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package nvme-cli. 2020-11-09T18:07:03.391 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../087-nvme-cli_1.5-1ubuntu1_amd64.deb ... 2020-11-09T18:07:03.429 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking nvme-cli (1.5-1ubuntu1) ... 2020-11-09T18:07:03.568 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up liboath0 (2.6.1-1) ... 2020-11-09T18:07:03.744 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-11-09T18:07:03.879 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2020-11-09T18:07:03.899 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python-attr. 2020-11-09T18:07:03.912 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../088-python-attr_17.4.0-2_all.deb ... 2020-11-09T18:07:03.949 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python-attr (17.4.0-2) ... 2020-11-09T18:07:04.130 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2020-11-09T18:07:04.231 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-setuptools (39.0.1-2) ... 2020-11-09T18:07:04.265 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python-funcsigs. 2020-11-09T18:07:04.277 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../089-python-funcsigs_1.0.2-4_all.deb ... 2020-11-09T18:07:04.332 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2020-11-09T18:07:04.591 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2020-11-09T18:07:04.642 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python-pluggy. 2020-11-09T18:07:04.656 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../090-python-pluggy_0.6.0-1_all.deb ... 2020-11-09T18:07:04.701 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python-pluggy (0.6.0-1) ... 2020-11-09T18:07:04.717 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-decorator (4.1.2-1) ... 2020-11-09T18:07:04.999 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-tz (2018.3-2) ... 2020-11-09T18:07:05.007 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python-py. 2020-11-09T18:07:05.020 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../091-python-py_1.5.2-1_all.deb ... 2020-11-09T18:07:05.058 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python-py (1.5.2-1) ... 2020-11-09T18:07:05.305 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2020-11-09T18:07:05.402 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python-pytest. 2020-11-09T18:07:05.414 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../092-python-pytest_3.3.2-2_all.deb ... 2020-11-09T18:07:05.456 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python-pytest (3.3.2-2) ... 2020-11-09T18:07:05.582 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2020-11-09T18:07:05.806 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-attr. 2020-11-09T18:07:05.818 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../093-python3-attr_17.4.0-2_all.deb ... 2020-11-09T18:07:05.856 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-attr (17.4.0-2) ... 2020-11-09T18:07:05.875 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-joblib (0.11-1) ... 2020-11-09T18:07:06.162 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-cycler. 2020-11-09T18:07:06.174 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../094-python3-cycler_0.10.0-1_all.deb ... 2020-11-09T18:07:06.213 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-cycler (0.10.0-1) ... 2020-11-09T18:07:06.274 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-routes (2.4.1-1) ... 2020-11-09T18:07:06.514 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-webencodings. 2020-11-09T18:07:06.526 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../095-python3-webencodings_0.5-2_all.deb ... 2020-11-09T18:07:06.564 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-webencodings (0.5-2) ... 2020-11-09T18:07:06.588 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2020-11-09T18:07:06.878 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-dateutil (2.6.1-1) ... 2020-11-09T18:07:06.891 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-html5lib. 2020-11-09T18:07:06.904 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../096-python3-html5lib_0.999999999-1_all.deb ... 2020-11-09T18:07:06.941 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2020-11-09T18:07:07.200 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.4) ... 2020-11-09T18:07:07.277 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-lxml:amd64. 2020-11-09T18:07:07.291 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../097-python3-lxml_4.2.1-1ubuntu0.1_amd64.deb ... 2020-11-09T18:07:07.327 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-11-09T18:07:07.588 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-fasteners (0.12.0-3) ... 2020-11-09T18:07:07.764 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-pyparsing. 2020-11-09T18:07:07.777 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../098-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2020-11-09T18:07:07.814 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2020-11-09T18:07:07.867 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up jq (1.5+dfsg-2) ... 2020-11-09T18:07:07.988 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2020-11-09T18:07:08.100 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-tz. 2020-11-09T18:07:08.114 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../099-python3-tz_2018.3-2_all.deb ... 2020-11-09T18:07:08.150 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-tz (2018.3-2) ... 2020-11-09T18:07:08.456 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-matplotlib. 2020-11-09T18:07:08.469 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../100-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2020-11-09T18:07:08.506 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2020-11-09T18:07:09.147 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-11-09T18:07:09.261 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-olefile. 2020-11-09T18:07:09.274 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../101-python3-olefile_0.45.1-1_all.deb ... 2020-11-09T18:07:09.311 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-olefile (0.45.1-1) ... 2020-11-09T18:07:09.468 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2020-11-09T18:07:09.614 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-pastescript. 2020-11-09T18:07:09.627 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../102-python3-pastescript_2.0.2-2_all.deb ... 2020-11-09T18:07:09.667 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2020-11-09T18:07:09.817 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2020-11-09T18:07:09.962 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pytest (3.3.2-2) ... 2020-11-09T18:07:10.012 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-pil:amd64. 2020-11-09T18:07:10.025 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../103-python3-pil_5.1.0-1ubuntu0.3_amd64.deb ... 2020-11-09T18:07:10.062 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-11-09T18:07:10.407 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-pluggy. 2020-11-09T18:07:10.422 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../104-python3-pluggy_0.6.0-1_all.deb ... 2020-11-09T18:07:10.457 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python-pytest (3.3.2-2) ... 2020-11-09T18:07:10.465 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2020-11-09T18:07:10.721 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2020-11-09T18:07:10.768 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-py. 2020-11-09T18:07:10.781 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../105-python3-py_1.5.2-1_all.deb ... 2020-11-09T18:07:10.818 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-py (1.5.2-1) ... 2020-11-09T18:07:10.838 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up xmlstarlet (1.6.1-2) ... 2020-11-09T18:07:10.958 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up librados2 (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:11.086 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-rsa (3.4.2-1) ... 2020-11-09T18:07:11.100 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-pyinotify. 2020-11-09T18:07:11.115 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../106-python3-pyinotify_0.9.6-1_all.deb ... 2020-11-09T18:07:11.150 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2020-11-09T18:07:11.385 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libcephfs2 (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:11.453 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-pytest. 2020-11-09T18:07:11.466 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../107-python3-pytest_3.3.2-2_all.deb ... 2020-11-09T18:07:11.490 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libcephfs-dev (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:11.503 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-pytest (3.3.2-2) ... 2020-11-09T18:07:11.599 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-rados (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:11.755 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2020-11-09T18:07:11.840 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-simplejson. 2020-11-09T18:07:11.854 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../108-python3-simplejson_3.13.2-1_amd64.deb ... 2020-11-09T18:07:11.898 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2020-11-09T18:07:12.052 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2020-11-09T18:07:12.213 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package python3-tk:amd64. 2020-11-09T18:07:12.227 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../109-python3-tk_3.6.9-1~18.04_amd64.deb ... 2020-11-09T18:07:12.271 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2020-11-09T18:07:12.351 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2020-11-09T18:07:12.667 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package radosgw. 2020-11-09T18:07:12.680 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../110-radosgw_16.0.0-7048-ga0560c8e-1bionic_amd64.deb ... 2020-11-09T18:07:12.681 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2020-11-09T18:07:12.717 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking radosgw (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:13.634 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-sklearn (0.19.1-3) ... 2020-11-09T18:07:13.787 INFO:teuthology.orchestra.run.smithi051.stdout:Selecting previously unselected package rbd-fuse. 2020-11-09T18:07:13.801 INFO:teuthology.orchestra.run.smithi051.stdout:Preparing to unpack .../111-rbd-fuse_16.0.0-7048-ga0560c8e-1bionic_amd64.deb ... 2020-11-09T18:07:13.845 INFO:teuthology.orchestra.run.smithi051.stdout:Unpacking rbd-fuse (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:14.132 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-logutils (0.3.3-5) ... 2020-11-09T18:07:14.420 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1) ... 2020-11-09T18:07:14.472 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2020-11-09T18:07:14.768 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libradosstriper1 (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:14.847 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2020-11-09T18:07:14.890 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-cephfs (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:14.981 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-11-09T18:07:15.168 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up librgw2 (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:15.296 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-fuse (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:15.516 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-11-09T18:07:15.516 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-11-09T18:07:15.617 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-11-09T18:07:15.776 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up librbd1 (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:15.885 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pecan (1.2.1-2) ... 2020-11-09T18:07:15.983 INFO:teuthology.orchestra.run.smithi078.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2020-11-09T18:07:16.016 INFO:teuthology.orchestra.run.smithi078.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2020-11-09T18:07:16.018 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2020-11-09T18:07:16.307 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-webencodings (0.5-2) ... 2020-11-09T18:07:16.322 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2020-11-09T18:07:16.583 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python-attr (17.4.0-2) ... 2020-11-09T18:07:16.784 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2020-11-09T18:07:16.910 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-py (1.5.2-1) ... 2020-11-09T18:07:17.000 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-mgr-modules-core (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:17.109 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pastescript (2.0.2-2) ... 2020-11-09T18:07:17.262 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-waitress (1.0.1-1) ... 2020-11-09T18:07:17.418 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-rgw (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:17.545 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up rbd-fuse (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:17.601 INFO:teuthology.orchestra.run.smithi051.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2020-11-09T18:07:17.671 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-rbd (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:17.797 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-common (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:17.906 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-olefile (0.45.1-1) ... 2020-11-09T18:07:18.196 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-ceph-argparse (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:18.304 INFO:teuthology.orchestra.run.smithi078.stdout:Adding group ceph....done 2020-11-09T18:07:18.483 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-tempita (0.5.2-2) ... 2020-11-09T18:07:18.675 INFO:teuthology.orchestra.run.smithi078.stdout:Adding system user ceph....done 2020-11-09T18:07:18.793 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2020-11-09T18:07:18.867 INFO:teuthology.orchestra.run.smithi078.stdout:Setting system user ceph properties....done 2020-11-09T18:07:18.919 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2020-11-09T18:07:19.010 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2020-11-09T18:07:19.218 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2020-11-09T18:07:19.522 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up radosgw (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:19.791 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-11-09T18:07:19.792 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-11-09T18:07:19.905 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2020-11-09T18:07:20.166 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-test (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:20.231 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2020-11-09T18:07:20.284 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-base (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:20.499 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python-py (1.5.2-1) ... 2020-11-09T18:07:20.578 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2020-11-09T18:07:20.701 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-simplejson (3.13.2-1) ... 2020-11-09T18:07:20.844 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-mgr (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:21.024 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-json-pointer (1.10-1) ... 2020-11-09T18:07:21.062 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-11-09T18:07:21.063 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-11-09T18:07:21.097 INFO:teuthology.orchestra.run.smithi051.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2020-11-09T18:07:21.331 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-mgr-diskprediction-local (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:21.334 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2020-11-09T18:07:21.460 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python-pluggy (0.6.0-1) ... 2020-11-09T18:07:21.474 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-osd (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:21.661 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2020-11-09T18:07:21.893 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-11-09T18:07:21.894 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-11-09T18:07:21.954 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2020-11-09T18:07:22.143 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-mgr-cephadm (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:22.261 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-mgr-rook (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:22.280 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-cycler (0.10.0-1) ... 2020-11-09T18:07:22.395 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-mds (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:22.565 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2020-11-09T18:07:22.619 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-11-09T18:07:22.620 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-11-09T18:07:22.888 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-mon (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:22.924 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-bs4 (4.6.0-1) ... 2020-11-09T18:07:23.113 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-11-09T18:07:23.114 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-11-09T18:07:23.254 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-pluggy (0.6.0-1) ... 2020-11-09T18:07:23.398 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-mgr-k8sevents (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:23.524 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-mgr-dashboard (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:23.547 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up cephadm (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:23.670 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:23.808 INFO:teuthology.orchestra.run.smithi078.stdout:Processing triggers for systemd (237-3ubuntu10.42) ... 2020-11-09T18:07:24.027 INFO:teuthology.orchestra.run.smithi078.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2020-11-09T18:07:24.173 INFO:teuthology.orchestra.run.smithi051.stdout:Adding system user cephadm....done 2020-11-09T18:07:24.337 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2020-11-09T18:07:24.663 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-11-09T18:07:24.789 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up nvme-cli (1.5-1ubuntu1) ... 2020-11-09T18:07:24.910 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2020-11-09T18:07:25.261 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-prettytable (0.7.2-3) ... 2020-11-09T18:07:25.562 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2020-11-09T18:07:25.688 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2020-11-09T18:07:26.040 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-ceph-common (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:26.075 INFO:teuthology.orchestra.run.smithi078.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2020-11-09T18:07:26.119 INFO:teuthology.orchestra.run.smithi078.stdout:ureadahead will be reprofiled on next reboot 2020-11-09T18:07:26.168 INFO:teuthology.orchestra.run.smithi078.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.3) ... 2020-11-09T18:07:26.349 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-monotonic (1.1-2) ... 2020-11-09T18:07:26.614 INFO:teuthology.orchestra.run.smithi078.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2020-11-09T18:07:26.617 DEBUG:teuthology.parallel:result is None 2020-11-09T18:07:26.633 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2020-11-09T18:07:26.751 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-11-09T18:07:27.094 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-html5lib (0.999999999-1) ... 2020-11-09T18:07:27.449 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-11-09T18:07:27.522 INFO:teuthology.orchestra.run.smithi051.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2020-11-09T18:07:27.554 INFO:teuthology.orchestra.run.smithi051.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2020-11-09T18:07:27.793 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-repoze.lru (0.7-1) ... 2020-11-09T18:07:28.102 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-11-09T18:07:28.412 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2020-11-09T18:07:28.756 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-attr (17.4.0-2) ... 2020-11-09T18:07:29.064 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python-funcsigs (1.0.2-4) ... 2020-11-09T18:07:29.273 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-11-09T18:07:29.688 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2020-11-09T18:07:29.777 INFO:teuthology.orchestra.run.smithi051.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2020-11-09T18:07:30.031 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up liboath0 (2.6.1-1) ... 2020-11-09T18:07:30.207 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-11-09T18:07:30.308 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2020-11-09T18:07:30.559 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2020-11-09T18:07:30.686 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-setuptools (39.0.1-2) ... 2020-11-09T18:07:31.087 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2020-11-09T18:07:31.214 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-decorator (4.1.2-1) ... 2020-11-09T18:07:31.511 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-tz (2018.3-2) ... 2020-11-09T18:07:31.832 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2020-11-09T18:07:32.129 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2020-11-09T18:07:32.475 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-joblib (0.11-1) ... 2020-11-09T18:07:32.848 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-routes (2.4.1-1) ... 2020-11-09T18:07:33.154 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2020-11-09T18:07:33.464 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-dateutil (2.6.1-1) ... 2020-11-09T18:07:33.784 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.4) ... 2020-11-09T18:07:34.141 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-fasteners (0.12.0-3) ... 2020-11-09T18:07:34.436 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up jq (1.5+dfsg-2) ... 2020-11-09T18:07:34.565 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2020-11-09T18:07:35.776 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-11-09T18:07:36.097 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2020-11-09T18:07:36.459 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2020-11-09T18:07:36.615 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-pytest (3.3.2-2) ... 2020-11-09T18:07:36.962 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python-pytest (3.3.2-2) ... 2020-11-09T18:07:37.223 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2020-11-09T18:07:37.326 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up xmlstarlet (1.6.1-2) ... 2020-11-09T18:07:37.454 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up librados2 (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:37.578 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-rsa (3.4.2-1) ... 2020-11-09T18:07:37.899 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up libcephfs2 (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:38.011 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up libcephfs-dev (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:38.132 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-rados (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:38.262 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2020-11-09T18:07:38.572 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2020-11-09T18:07:38.895 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2020-11-09T18:07:39.229 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2020-11-09T18:07:40.203 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-sklearn (0.19.1-3) ... 2020-11-09T18:07:41.078 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2020-11-09T18:07:41.405 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up libradosstriper1 (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:41.529 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-cephfs (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:41.841 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up librgw2 (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:41.947 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up ceph-fuse (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:42.170 INFO:teuthology.orchestra.run.smithi051.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-11-09T18:07:42.171 INFO:teuthology.orchestra.run.smithi051.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-11-09T18:07:42.434 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up librbd1 (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:42.554 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-pecan (1.2.1-2) ... 2020-11-09T18:07:42.645 INFO:teuthology.orchestra.run.smithi051.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2020-11-09T18:07:42.669 INFO:teuthology.orchestra.run.smithi051.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2020-11-09T18:07:42.961 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2020-11-09T18:07:43.621 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up ceph-mgr-modules-core (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:43.747 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-pastescript (2.0.2-2) ... 2020-11-09T18:07:44.065 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-rgw (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:44.167 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up rbd-fuse (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:44.267 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up python3-rbd (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:44.377 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up ceph-common (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:44.919 INFO:teuthology.orchestra.run.smithi051.stdout:Adding group ceph....done 2020-11-09T18:07:45.297 INFO:teuthology.orchestra.run.smithi051.stdout:Adding system user ceph....done 2020-11-09T18:07:45.481 INFO:teuthology.orchestra.run.smithi051.stdout:Setting system user ceph properties....done 2020-11-09T18:07:45.637 INFO:teuthology.orchestra.run.smithi051.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2020-11-09T18:07:45.863 INFO:teuthology.orchestra.run.smithi051.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2020-11-09T18:07:46.463 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up radosgw (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:46.736 INFO:teuthology.orchestra.run.smithi051.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-11-09T18:07:46.736 INFO:teuthology.orchestra.run.smithi051.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-11-09T18:07:47.106 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up ceph-test (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:47.234 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up ceph-base (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:47.557 INFO:teuthology.orchestra.run.smithi051.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2020-11-09T18:07:47.802 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up ceph-mgr (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:48.021 INFO:teuthology.orchestra.run.smithi051.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-11-09T18:07:48.021 INFO:teuthology.orchestra.run.smithi051.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-11-09T18:07:48.280 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up ceph-mgr-diskprediction-local (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:48.422 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up ceph-osd (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:48.860 INFO:teuthology.orchestra.run.smithi051.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-11-09T18:07:48.860 INFO:teuthology.orchestra.run.smithi051.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-11-09T18:07:49.127 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up ceph-mgr-cephadm (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:49.261 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up ceph-mgr-rook (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:49.404 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up ceph-mds (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:49.631 INFO:teuthology.orchestra.run.smithi051.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-11-09T18:07:49.631 INFO:teuthology.orchestra.run.smithi051.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-11-09T18:07:49.897 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up ceph-mon (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:50.118 INFO:teuthology.orchestra.run.smithi051.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-11-09T18:07:50.118 INFO:teuthology.orchestra.run.smithi051.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-11-09T18:07:50.386 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up ceph-mgr-k8sevents (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:50.516 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up ceph-mgr-dashboard (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:50.651 INFO:teuthology.orchestra.run.smithi051.stdout:Setting up ceph (16.0.0-7048-ga0560c8e-1bionic) ... 2020-11-09T18:07:50.752 INFO:teuthology.orchestra.run.smithi051.stdout:Processing triggers for systemd (237-3ubuntu10.42) ... 2020-11-09T18:07:50.944 INFO:teuthology.orchestra.run.smithi051.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2020-11-09T18:07:53.033 INFO:teuthology.orchestra.run.smithi051.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2020-11-09T18:07:53.084 INFO:teuthology.orchestra.run.smithi051.stdout:ureadahead will be reprofiled on next reboot 2020-11-09T18:07:53.136 INFO:teuthology.orchestra.run.smithi051.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.3) ... 2020-11-09T18:07:53.577 INFO:teuthology.orchestra.run.smithi051.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2020-11-09T18:07:53.579 DEBUG:teuthology.parallel:result is None 2020-11-09T18:07:53.580 INFO:teuthology.orchestra.run.smithi051:> sudo lsb_release -is 2020-11-09T18:07:53.632 INFO:teuthology.orchestra.run.smithi051.stdout:Ubuntu 2020-11-09T18:07:53.637 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-11-09T18:07:53.638 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=a0560c8e5c6c33c4655e9d2a8b022793ae57419f 2020-11-09T18:07:53.820 INFO:teuthology.orchestra.run.smithi051:> dpkg-query -W -f '${Version}' ceph 2020-11-09T18:07:53.843 INFO:teuthology.orchestra.run.smithi051.stdout:16.0.0-7048-ga0560c8e-1bionic 2020-11-09T18:07:53.844 INFO:teuthology.packaging:The installed version of ceph is 16.0.0-7048-ga0560c8e-1bionic 2020-11-09T18:07:53.844 INFO:teuthology.task.install:The correct ceph version 16.0.0-7048-ga0560c8e-1bionic is installed. 2020-11-09T18:07:53.845 INFO:teuthology.orchestra.run.smithi078:> sudo lsb_release -is 2020-11-09T18:07:53.903 INFO:teuthology.orchestra.run.smithi078.stdout:Ubuntu 2020-11-09T18:07:53.908 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-11-09T18:07:53.908 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=a0560c8e5c6c33c4655e9d2a8b022793ae57419f 2020-11-09T18:07:54.074 INFO:teuthology.orchestra.run.smithi078:> dpkg-query -W -f '${Version}' ceph 2020-11-09T18:07:54.099 INFO:teuthology.orchestra.run.smithi078.stdout:16.0.0-7048-ga0560c8e-1bionic 2020-11-09T18:07:54.100 INFO:teuthology.packaging:The installed version of ceph is 16.0.0-7048-ga0560c8e-1bionic 2020-11-09T18:07:54.100 INFO:teuthology.task.install:The correct ceph version 16.0.0-7048-ga0560c8e-1bionic is installed. 2020-11-09T18:07:54.101 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2020-11-09T18:07:54.101 INFO:teuthology.orchestra.run.smithi051:> set -ex 2020-11-09T18:07:54.101 INFO:teuthology.orchestra.run.smithi051:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-11-09T18:07:54.106 INFO:teuthology.orchestra.run.smithi051.stderr:+ sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-11-09T18:07:54.117 INFO:teuthology.orchestra.run.smithi051.stderr:24+1 records in 2020-11-09T18:07:54.117 INFO:teuthology.orchestra.run.smithi051.stderr:24+1 records out 2020-11-09T18:07:54.118 INFO:teuthology.orchestra.run.smithi051.stderr:12667 bytes (13 kB, 12 KiB) copied, 0.000216877 s, 58.4 MB/s 2020-11-09T18:07:54.118 INFO:teuthology.orchestra.run.smithi078:> set -ex 2020-11-09T18:07:54.119 INFO:teuthology.orchestra.run.smithi078:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-11-09T18:07:54.153 INFO:teuthology.orchestra.run.smithi078.stderr:+ sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-11-09T18:07:54.162 INFO:teuthology.orchestra.run.smithi078.stderr:24+1 records in 2020-11-09T18:07:54.162 INFO:teuthology.orchestra.run.smithi078.stderr:24+1 records out 2020-11-09T18:07:54.162 INFO:teuthology.orchestra.run.smithi078.stderr:12667 bytes (13 kB, 12 KiB) copied, 0.000218281 s, 58.0 MB/s 2020-11-09T18:07:54.163 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2020-11-09T18:07:54.163 INFO:teuthology.orchestra.run.smithi051:> set -ex 2020-11-09T18:07:54.164 INFO:teuthology.orchestra.run.smithi051:> sudo dd of=/usr/bin/daemon-helper 2020-11-09T18:07:54.169 INFO:teuthology.orchestra.run.smithi051.stderr:+ sudo dd of=/usr/bin/daemon-helper 2020-11-09T18:07:54.180 INFO:teuthology.orchestra.run.smithi051.stderr:5+1 records in 2020-11-09T18:07:54.180 INFO:teuthology.orchestra.run.smithi051.stderr:5+1 records out 2020-11-09T18:07:54.180 INFO:teuthology.orchestra.run.smithi051.stderr:3058 bytes (3.1 kB, 3.0 KiB) copied, 0.000104236 s, 29.3 MB/s 2020-11-09T18:07:54.181 INFO:teuthology.orchestra.run.smithi051:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-11-09T18:07:54.235 INFO:teuthology.orchestra.run.smithi078:> set -ex 2020-11-09T18:07:54.236 INFO:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/daemon-helper 2020-11-09T18:07:54.240 INFO:teuthology.orchestra.run.smithi078.stderr:+ sudo dd of=/usr/bin/daemon-helper 2020-11-09T18:07:54.249 INFO:teuthology.orchestra.run.smithi078.stderr:5+1 records in 2020-11-09T18:07:54.250 INFO:teuthology.orchestra.run.smithi078.stderr:5+1 records out 2020-11-09T18:07:54.250 INFO:teuthology.orchestra.run.smithi078.stderr:3058 bytes (3.1 kB, 3.0 KiB) copied, 9.7103e-05 s, 31.5 MB/s 2020-11-09T18:07:54.251 INFO:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-11-09T18:07:54.308 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2020-11-09T18:07:54.308 INFO:teuthology.orchestra.run.smithi051:> set -ex 2020-11-09T18:07:54.309 INFO:teuthology.orchestra.run.smithi051:> sudo dd of=/usr/bin/adjust-ulimits 2020-11-09T18:07:54.313 INFO:teuthology.orchestra.run.smithi051.stderr:+ sudo dd of=/usr/bin/adjust-ulimits 2020-11-09T18:07:54.322 INFO:teuthology.orchestra.run.smithi051.stderr:0+1 records in 2020-11-09T18:07:54.323 INFO:teuthology.orchestra.run.smithi051.stderr:0+1 records out 2020-11-09T18:07:54.323 INFO:teuthology.orchestra.run.smithi051.stderr:299 bytes copied, 6.906e-05 s, 4.3 MB/s 2020-11-09T18:07:54.324 INFO:teuthology.orchestra.run.smithi051:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-11-09T18:07:54.379 INFO:teuthology.orchestra.run.smithi078:> set -ex 2020-11-09T18:07:54.380 INFO:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/adjust-ulimits 2020-11-09T18:07:54.385 INFO:teuthology.orchestra.run.smithi078.stderr:+ sudo dd of=/usr/bin/adjust-ulimits 2020-11-09T18:07:54.394 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records in 2020-11-09T18:07:54.394 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records out 2020-11-09T18:07:54.395 INFO:teuthology.orchestra.run.smithi078.stderr:299 bytes copied, 6.6427e-05 s, 4.5 MB/s 2020-11-09T18:07:54.395 INFO:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-11-09T18:07:54.452 INFO:teuthology.run_tasks:Running task ceph... 2020-11-09T18:07:54.519 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2020-11-09T18:07:54.520 INFO:teuthology.orchestra.run.smithi051:> sudo chmod 777 /var/log/ceph 2020-11-09T18:07:54.523 INFO:teuthology.orchestra.run.smithi078:> sudo chmod 777 /var/log/ceph 2020-11-09T18:07:54.538 INFO:tasks.ceph:Disabling ceph logrotate... 2020-11-09T18:07:54.538 INFO:teuthology.orchestra.run.smithi051:> sudo rm -f -- /etc/logrotate.d/ceph 2020-11-09T18:07:54.578 INFO:teuthology.orchestra.run.smithi078:> sudo rm -f -- /etc/logrotate.d/ceph 2020-11-09T18:07:54.595 INFO:tasks.ceph:Creating extra log directories... 2020-11-09T18:07:54.595 INFO:teuthology.orchestra.run.smithi051:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-11-09T18:07:54.635 INFO:teuthology.orchestra.run.smithi078:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-11-09T18:07:54.662 INFO:tasks.ceph:Creating ceph cluster c1... 2020-11-09T18:07:54.662 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(PG_AVAILABILITY\\)'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2020-11-09T18:07:54.663 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/dang-2020-11-09_12:38:28-rgw-wip-dang-webdir-empty-object-distro-basic-smithi/5605822', 'branch': 'wip-dang-webdir-empty-object', 'description': 'rgw/multisite/{clusters frontend/civetweb omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '5605822', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'dang-2020-11-09_12:38:28-rgw-wip-dang-webdir-empty-object-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'ubuntu', 'overrides': {'admin_socket': {'branch': 'wip-dang-webdir-empty-object'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'a0560c8e5c6c33c4655e9d2a8b022793ae57419f', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': 'a0560c8e5c6c33c4655e9d2a8b022793ae57419f'}}, 'rgw': {'compression type': 'random', 'frontend': 'civetweb'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'workunit': {'branch': 'wip-dang-webdir-empty-object', 'sha1': 'a0560c8e5c6c33c4655e9d2a8b022793ae57419f'}}, 'owner': 'scheduled_dang@sidious.arb.redhat.com', 'priority': 100, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': 'a0560c8e5c6c33c4655e9d2a8b022793ae57419f', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-dang-webdir-empty-object', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_wip-dang-webdir-empty-object/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'a0560c8e5c6c33c4655e9d2a8b022793ae57419f', 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (2, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'a0560c8e5c6c33c4655e9d2a8b022793ae57419f', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'tube': 'smithi', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.7071', 'targets': {'smithi051.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDChRn9ME/9o1R6oGob2yIvGJjZ3e8odesSjGsarWb9ZjpSVVq1CLSG0gjPIQwGH3lAQOoqsxc3hyAemED82m5rAOpsoeCiYN2zA5MTNVqv+oHY283y2tPAnLGcjMMIFpe/YoYBVUDoH2ppv4cpoKAD/WX31GH84Z7mXBopdTgcDZfRU6gH6YeE0/mj34rUy3Z/6oxwrNkhiMAJADRMiga/xTTAKFJQaAc4UT0JVWIp7yBdY6AVwj24vC7x35EXpJwmDwMfWAMaHxFoy/uJMazzWPyw0D7AQp857+92jl28ah1hwXTyhl4G0lqxhxX54tNsit+xwsOWwuqJpNPjUW0L', 'smithi078.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEXAj7zwy9ounBiobhGxgHVGG0io6UH6ZgSL8GxSgCz5Xjm6d8yZtJHCZRXwzH33gb18kSLU4Pc2qngB+l7qc0UMJLU31PvCo0Nz1/x2na6haA1A0UmPtd45BOsT5v1D5taYl1dN1y26UqxxeDj0NZzpgSuF1EdSKAt75H5cBJt7mu1UlDYcn0cPnsINXRm29Bp2e4X7hyj21y7QfQL6RTa3+/8T7de4l9lhjUqV/qZRepvYWmoB9pg6U0bEHgnvkS8UQefJXYMS++teARGcP1hY3KBciTDCF+WrxHkNUNyhmRUaM3wxvILElcgkVwK4vPp+AQHJ5OmtEltNe6t/Zh'}} 2020-11-09T18:07:54.663 INFO:teuthology.orchestra.run.smithi051:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2020-11-09T18:07:54.690 INFO:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2020-11-09T18:07:54.699 INFO:teuthology.orchestra.run.smithi051:> sudo install -d -m0777 -- /var/run/ceph 2020-11-09T18:07:54.738 INFO:teuthology.orchestra.run.smithi078:> sudo install -d -m0777 -- /var/run/ceph 2020-11-09T18:07:54.756 INFO:teuthology.orchestra.run.smithi051:> set -ex 2020-11-09T18:07:54.757 INFO:teuthology.orchestra.run.smithi051:> dd if=/scratch_devs of=/dev/stdout 2020-11-09T18:07:54.797 INFO:teuthology.orchestra.run.smithi051.stderr:+ dd if=/scratch_devs of=/dev/stdout 2020-11-09T18:07:54.798 INFO:teuthology.orchestra.run.smithi051.stdout:/dev/vg_nvme/lv_1 2020-11-09T18:07:54.798 INFO:teuthology.orchestra.run.smithi051.stdout:/dev/vg_nvme/lv_2 2020-11-09T18:07:54.798 INFO:teuthology.orchestra.run.smithi051.stdout:/dev/vg_nvme/lv_3 2020-11-09T18:07:54.799 INFO:teuthology.orchestra.run.smithi051.stdout:/dev/vg_nvme/lv_4 2020-11-09T18:07:54.799 INFO:teuthology.orchestra.run.smithi051.stderr:0+1 records in 2020-11-09T18:07:54.799 INFO:teuthology.orchestra.run.smithi051.stderr:0+1 records out 2020-11-09T18:07:54.799 INFO:teuthology.orchestra.run.smithi051.stderr:72 bytes copied, 0.00017115 s, 421 kB/s 2020-11-09T18:07:54.799 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2020-11-09T18:07:54.800 INFO:teuthology.orchestra.run.smithi051:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-11-09T18:07:54.847 INFO:teuthology.orchestra.run.smithi051.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2020-11-09T18:07:54.847 INFO:teuthology.orchestra.run.smithi051.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-09T18:07:54.848 INFO:teuthology.orchestra.run.smithi051.stdout:Device: 6h/6d Inode: 341 Links: 1 2020-11-09T18:07:54.848 INFO:teuthology.orchestra.run.smithi051.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-09T18:07:54.848 INFO:teuthology.orchestra.run.smithi051.stdout:Access: 2020-11-09 18:05:12.592264263 +0000 2020-11-09T18:07:54.848 INFO:teuthology.orchestra.run.smithi051.stdout:Modify: 2020-11-09 18:05:12.476264264 +0000 2020-11-09T18:07:54.848 INFO:teuthology.orchestra.run.smithi051.stdout:Change: 2020-11-09 18:05:12.476264264 +0000 2020-11-09T18:07:54.848 INFO:teuthology.orchestra.run.smithi051.stdout: Birth: - 2020-11-09T18:07:54.857 INFO:teuthology.orchestra.run.smithi051.stderr:1+0 records in 2020-11-09T18:07:54.857 INFO:teuthology.orchestra.run.smithi051.stderr:1+0 records out 2020-11-09T18:07:54.858 INFO:teuthology.orchestra.run.smithi051.stderr:512 bytes copied, 0.000407754 s, 1.3 MB/s 2020-11-09T18:07:54.862 INFO:teuthology.orchestra.run.smithi051:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-11-09T18:07:54.911 INFO:teuthology.orchestra.run.smithi051.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2020-11-09T18:07:54.912 INFO:teuthology.orchestra.run.smithi051.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-09T18:07:54.912 INFO:teuthology.orchestra.run.smithi051.stdout:Device: 6h/6d Inode: 348 Links: 1 2020-11-09T18:07:54.912 INFO:teuthology.orchestra.run.smithi051.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-09T18:07:54.912 INFO:teuthology.orchestra.run.smithi051.stdout:Access: 2020-11-09 18:05:12.840264260 +0000 2020-11-09T18:07:54.912 INFO:teuthology.orchestra.run.smithi051.stdout:Modify: 2020-11-09 18:05:12.740264261 +0000 2020-11-09T18:07:54.912 INFO:teuthology.orchestra.run.smithi051.stdout:Change: 2020-11-09 18:05:12.740264261 +0000 2020-11-09T18:07:54.913 INFO:teuthology.orchestra.run.smithi051.stdout: Birth: - 2020-11-09T18:07:54.921 INFO:teuthology.orchestra.run.smithi051.stderr:1+0 records in 2020-11-09T18:07:54.921 INFO:teuthology.orchestra.run.smithi051.stderr:1+0 records out 2020-11-09T18:07:54.921 INFO:teuthology.orchestra.run.smithi051.stderr:512 bytes copied, 0.000421963 s, 1.2 MB/s 2020-11-09T18:07:54.926 INFO:teuthology.orchestra.run.smithi051:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-11-09T18:07:54.976 INFO:teuthology.orchestra.run.smithi051.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2020-11-09T18:07:55.028 INFO:teuthology.orchestra.run.smithi051.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-09T18:07:55.029 INFO:teuthology.orchestra.run.smithi051.stdout:Device: 6h/6d Inode: 354 Links: 1 2020-11-09T18:07:55.029 INFO:teuthology.orchestra.run.smithi051.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-09T18:07:55.029 INFO:teuthology.orchestra.run.smithi051.stdout:Access: 2020-11-09 18:05:13.096264257 +0000 2020-11-09T18:07:55.029 INFO:teuthology.orchestra.run.smithi051.stdout:Modify: 2020-11-09 18:05:12.980264259 +0000 2020-11-09T18:07:55.029 INFO:teuthology.orchestra.run.smithi051.stdout:Change: 2020-11-09 18:05:12.980264259 +0000 2020-11-09T18:07:55.029 INFO:teuthology.orchestra.run.smithi051.stdout: Birth: - 2020-11-09T18:07:55.030 INFO:teuthology.orchestra.run.smithi051.stderr:1+0 records in 2020-11-09T18:07:55.031 INFO:teuthology.orchestra.run.smithi051.stderr:1+0 records out 2020-11-09T18:07:55.031 INFO:teuthology.orchestra.run.smithi051.stderr:512 bytes copied, 0.000395624 s, 1.3 MB/s 2020-11-09T18:07:55.031 INFO:teuthology.orchestra.run.smithi051:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-11-09T18:07:55.079 INFO:teuthology.orchestra.run.smithi051.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2020-11-09T18:07:55.079 INFO:teuthology.orchestra.run.smithi051.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-09T18:07:55.080 INFO:teuthology.orchestra.run.smithi051.stdout:Device: 6h/6d Inode: 360 Links: 1 2020-11-09T18:07:55.080 INFO:teuthology.orchestra.run.smithi051.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-09T18:07:55.080 INFO:teuthology.orchestra.run.smithi051.stdout:Access: 2020-11-09 18:05:13.360264255 +0000 2020-11-09T18:07:55.080 INFO:teuthology.orchestra.run.smithi051.stdout:Modify: 2020-11-09 18:05:13.240264256 +0000 2020-11-09T18:07:55.080 INFO:teuthology.orchestra.run.smithi051.stdout:Change: 2020-11-09 18:05:13.240264256 +0000 2020-11-09T18:07:55.080 INFO:teuthology.orchestra.run.smithi051.stdout: Birth: - 2020-11-09T18:07:55.089 INFO:teuthology.orchestra.run.smithi051.stderr:1+0 records in 2020-11-09T18:07:55.089 INFO:teuthology.orchestra.run.smithi051.stderr:1+0 records out 2020-11-09T18:07:55.089 INFO:teuthology.orchestra.run.smithi051.stderr:512 bytes copied, 0.000322728 s, 1.6 MB/s 2020-11-09T18:07:55.093 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-09T18:07:55.093 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi051.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2020-11-09T18:07:55.094 INFO:tasks.ceph:Generating config... 2020-11-09T18:07:55.095 INFO:tasks.ceph:[client] debug rgw = 20 2020-11-09T18:07:55.096 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2020-11-09T18:07:55.096 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2020-11-09T18:07:55.096 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-11-09T18:07:55.097 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2020-11-09T18:07:55.097 INFO:tasks.ceph:[client] rgw data log num shards = 4 2020-11-09T18:07:55.097 INFO:tasks.ceph:[client] rgw md log max shards = 4 2020-11-09T18:07:55.097 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2020-11-09T18:07:55.098 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2020-11-09T18:07:55.098 INFO:tasks.ceph:[client] setgroup = ceph 2020-11-09T18:07:55.098 INFO:tasks.ceph:[client] setuser = ceph 2020-11-09T18:07:55.099 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2020-11-09T18:07:55.099 INFO:tasks.ceph:[mgr] debug mgr = 20 2020-11-09T18:07:55.099 INFO:tasks.ceph:[mgr] debug ms = 1 2020-11-09T18:07:55.100 INFO:tasks.ceph:[mon] debug mon = 20 2020-11-09T18:07:55.100 INFO:tasks.ceph:[mon] debug ms = 1 2020-11-09T18:07:55.100 INFO:tasks.ceph:[mon] debug paxos = 20 2020-11-09T18:07:55.101 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2020-11-09T18:07:55.101 INFO:tasks.ceph:[osd] debug filestore = 20 2020-11-09T18:07:55.101 INFO:tasks.ceph:[osd] debug journal = 20 2020-11-09T18:07:55.102 INFO:tasks.ceph:[osd] debug ms = 20 2020-11-09T18:07:55.103 INFO:tasks.ceph:[osd] debug osd = 25 2020-11-09T18:07:55.103 INFO:tasks.ceph:[osd] osd fast shutdown = False 2020-11-09T18:07:55.103 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2020-11-09T18:07:55.103 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2020-11-09T18:07:55.104 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2020-11-09T18:07:55.104 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2020-11-09T18:07:55.104 INFO:tasks.ceph:Setting up c1.mon.a... 2020-11-09T18:07:55.104 INFO:teuthology.orchestra.run.smithi051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2020-11-09T18:07:55.169 INFO:teuthology.orchestra.run.smithi051.stdout:creating /etc/ceph/c1.keyring 2020-11-09T18:07:55.171 INFO:teuthology.orchestra.run.smithi051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2020-11-09T18:07:55.249 INFO:teuthology.orchestra.run.smithi051:> sudo chmod 0644 /etc/ceph/c1.keyring 2020-11-09T18:07:55.306 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.51')] 2020-11-09T18:07:55.307 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon host': '172.21.15.51', 'osd heartbeat grace': 40}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'mon osd crush smoke test': False}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2020-11-09T18:07:55.308 INFO:teuthology.orchestra.run.smithi051:> set -ex 2020-11-09T18:07:55.308 INFO:teuthology.orchestra.run.smithi051:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2020-11-09T18:07:55.353 INFO:teuthology.orchestra.run.smithi051.stderr:+ dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2020-11-09T18:07:55.354 INFO:teuthology.orchestra.run.smithi051.stderr:6+1 records in 2020-11-09T18:07:55.355 INFO:teuthology.orchestra.run.smithi051.stderr:6+1 records out 2020-11-09T18:07:55.355 INFO:teuthology.orchestra.run.smithi051.stderr:3100 bytes (3.1 kB, 3.0 KiB) copied, 0.0001066 s, 29.1 MB/s 2020-11-09T18:07:55.355 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.51 --print /home/ubuntu/cephtest/c1.monmap 2020-11-09T18:07:55.429 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:07:55.430 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:07:55.430 INFO:teuthology.orchestra.run.smithi051.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2020-11-09T18:07:55.430 INFO:teuthology.orchestra.run.smithi051.stdout:monmaptool: generated fsid f8cfda89-96d9-4149-8cfc-db3ea75def88 2020-11-09T18:07:55.430 INFO:teuthology.orchestra.run.smithi051.stdout:epoch 0 2020-11-09T18:07:55.430 INFO:teuthology.orchestra.run.smithi051.stdout:fsid f8cfda89-96d9-4149-8cfc-db3ea75def88 2020-11-09T18:07:55.430 INFO:teuthology.orchestra.run.smithi051.stdout:last_changed 2020-11-09T18:07:55.431831+0000 2020-11-09T18:07:55.431 INFO:teuthology.orchestra.run.smithi051.stdout:created 2020-11-09T18:07:55.431831+0000 2020-11-09T18:07:55.431 INFO:teuthology.orchestra.run.smithi051.stdout:min_mon_release 0 (unknown) 2020-11-09T18:07:55.431 INFO:teuthology.orchestra.run.smithi051.stdout:election_strategy: 1 2020-11-09T18:07:55.431 INFO:teuthology.orchestra.run.smithi051.stdout:0: [v2:172.21.15.51:3300/0,v1:172.21.15.51:6789/0] mon.a 2020-11-09T18:07:55.432 INFO:teuthology.orchestra.run.smithi051.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2020-11-09T18:07:55.433 INFO:teuthology.orchestra.run.smithi051:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2020-11-09T18:07:55.483 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID f8cfda89-96d9-4149-8cfc-db3ea75def88... 2020-11-09T18:07:55.485 INFO:teuthology.orchestra.run.smithi051:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2020-11-09T18:07:55.527 INFO:teuthology.orchestra.run.smithi078:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2020-11-09T18:07:55.557 INFO:teuthology.orchestra.run.smithi051.stdout:[global] 2020-11-09T18:07:55.558 INFO:teuthology.orchestra.run.smithi051.stdout: chdir = "" 2020-11-09T18:07:55.558 INFO:teuthology.orchestra.run.smithi051.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-09T18:07:55.558 INFO:teuthology.orchestra.run.smithi051.stdout: auth supported = cephx 2020-11-09T18:07:55.558 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:07:55.558 INFO:teuthology.orchestra.run.smithi051.stdout: filestore xattr use omap = true 2020-11-09T18:07:55.559 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:07:55.559 INFO:teuthology.orchestra.run.smithi051.stdout: mon clock drift allowed = 1.000 2020-11-09T18:07:55.559 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:07:55.559 INFO:teuthology.orchestra.run.smithi051.stdout: osd crush chooseleaf type = 0 2020-11-09T18:07:55.559 INFO:teuthology.orchestra.run.smithi051.stdout: auth debug = true 2020-11-09T18:07:55.559 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:07:55.559 INFO:teuthology.orchestra.run.smithi051.stdout: ms die on old message = true 2020-11-09T18:07:55.560 INFO:teuthology.orchestra.run.smithi051.stdout: ms die on bug = true 2020-11-09T18:07:55.560 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:07:55.560 INFO:teuthology.orchestra.run.smithi051.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-09T18:07:55.560 INFO:teuthology.orchestra.run.smithi051.stdout: mon pg warn max object skew = 0 2020-11-09T18:07:55.560 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:07:55.560 INFO:teuthology.orchestra.run.smithi051.stdout: # disable pg_autoscaler by default for new pools 2020-11-09T18:07:55.560 INFO:teuthology.orchestra.run.smithi051.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-09T18:07:55.560 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:07:55.561 INFO:teuthology.orchestra.run.smithi051.stdout: osd pool default size = 2 2020-11-09T18:07:55.561 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:07:55.561 INFO:teuthology.orchestra.run.smithi051.stdout: mon osd allow primary affinity = true 2020-11-09T18:07:55.561 INFO:teuthology.orchestra.run.smithi051.stdout: mon osd allow pg remap = true 2020-11-09T18:07:55.561 INFO:teuthology.orchestra.run.smithi051.stdout: mon warn on legacy crush tunables = false 2020-11-09T18:07:55.561 INFO:teuthology.orchestra.run.smithi051.stdout: mon warn on crush straw calc version zero = false 2020-11-09T18:07:55.561 INFO:teuthology.orchestra.run.smithi051.stdout: mon warn on no sortbitwise = false 2020-11-09T18:07:55.562 INFO:teuthology.orchestra.run.smithi051.stdout: mon warn on osd down out interval zero = false 2020-11-09T18:07:55.563 INFO:teuthology.orchestra.run.smithi051.stdout: mon warn on too few osds = false 2020-11-09T18:07:55.563 INFO:teuthology.orchestra.run.smithi051.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-09T18:07:55.563 INFO:teuthology.orchestra.run.smithi051.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-09T18:07:55.563 INFO:teuthology.orchestra.run.smithi051.stdout: mon_allow_pool_size_one = true 2020-11-09T18:07:55.563 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:07:55.564 INFO:teuthology.orchestra.run.smithi051.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-09T18:07:55.564 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:07:55.564 INFO:teuthology.orchestra.run.smithi051.stdout: osd default data pool replay window = 5 2020-11-09T18:07:55.564 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:07:55.564 INFO:teuthology.orchestra.run.smithi051.stdout: mon allow pool delete = true 2020-11-09T18:07:55.564 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:07:55.564 INFO:teuthology.orchestra.run.smithi051.stdout: mon cluster log file level = debug 2020-11-09T18:07:55.565 INFO:teuthology.orchestra.run.smithi051.stdout: debug asserts on shutdown = true 2020-11-09T18:07:55.565 INFO:teuthology.orchestra.run.smithi051.stdout: mon host = 172.21.15.51 2020-11-09T18:07:55.565 INFO:teuthology.orchestra.run.smithi051.stdout: osd heartbeat grace = 40 2020-11-09T18:07:55.565 INFO:teuthology.orchestra.run.smithi051.stdout: fsid = f8cfda89-96d9-4149-8cfc-db3ea75def88 2020-11-09T18:07:55.566 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:07:55.566 INFO:teuthology.orchestra.run.smithi051.stdout:[osd] 2020-11-09T18:07:55.566 INFO:teuthology.orchestra.run.smithi051.stdout: osd journal size = 100 2020-11-09T18:07:55.566 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:07:55.566 INFO:teuthology.orchestra.run.smithi051.stdout: osd scrub load threshold = 5.0 2020-11-09T18:07:55.566 INFO:teuthology.orchestra.run.smithi051.stdout: osd scrub max interval = 600 2020-11-09T18:07:55.567 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:07:55.567 INFO:teuthology.orchestra.run.smithi051.stdout: osd recover clone overlap = true 2020-11-09T18:07:55.567 INFO:teuthology.orchestra.run.smithi051.stdout: osd recovery max chunk = 1048576 2020-11-09T18:07:55.567 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:07:55.567 INFO:teuthology.orchestra.run.smithi051.stdout: osd debug shutdown = true 2020-11-09T18:07:55.567 INFO:teuthology.orchestra.run.smithi051.stdout: osd debug op order = true 2020-11-09T18:07:55.567 INFO:teuthology.orchestra.run.smithi051.stdout: osd debug verify stray on activate = true 2020-11-09T18:07:55.568 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:07:55.568 INFO:teuthology.orchestra.run.smithi051.stdout: osd open classes on start = true 2020-11-09T18:07:55.568 INFO:teuthology.orchestra.run.smithi051.stdout: osd debug pg log writeout = true 2020-11-09T18:07:55.568 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:07:55.568 INFO:teuthology.orchestra.run.smithi051.stdout: osd deep scrub update digest min age = 30 2020-11-09T18:07:55.569 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:07:55.569 INFO:teuthology.orchestra.run.smithi051.stdout: osd map max advance = 10 2020-11-09T18:07:55.569 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:07:55.569 INFO:teuthology.orchestra.run.smithi051.stdout: journal zero on create = true 2020-11-09T18:07:55.569 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:07:55.569 INFO:teuthology.orchestra.run.smithi051.stdout: filestore ondisk finisher threads = 3 2020-11-09T18:07:55.569 INFO:teuthology.orchestra.run.smithi051.stdout: filestore apply finisher threads = 3 2020-11-09T18:07:55.570 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:07:55.570 INFO:teuthology.orchestra.run.smithi051.stdout: bdev debug aio = true 2020-11-09T18:07:55.570 INFO:teuthology.orchestra.run.smithi051.stdout: osd debug misdirected ops = true 2020-11-09T18:07:55.570 INFO:teuthology.orchestra.run.smithi051.stdout: debug filestore = 20 2020-11-09T18:07:55.570 INFO:teuthology.orchestra.run.smithi051.stdout: debug journal = 20 2020-11-09T18:07:55.570 INFO:teuthology.orchestra.run.smithi051.stdout: debug ms = 20 2020-11-09T18:07:55.571 INFO:teuthology.orchestra.run.smithi051.stdout: debug osd = 25 2020-11-09T18:07:55.571 INFO:teuthology.orchestra.run.smithi051.stdout: osd fast shutdown = False 2020-11-09T18:07:55.571 INFO:teuthology.orchestra.run.smithi051.stdout: rocksdb delete range threshold = 0 2020-11-09T18:07:55.571 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:07:55.571 INFO:teuthology.orchestra.run.smithi051.stdout:[mgr] 2020-11-09T18:07:55.571 INFO:teuthology.orchestra.run.smithi051.stdout: debug ms = 1 2020-11-09T18:07:55.572 INFO:teuthology.orchestra.run.smithi051.stdout: debug mgr = 20 2020-11-09T18:07:55.572 INFO:teuthology.orchestra.run.smithi051.stdout: debug mon = 20 2020-11-09T18:07:55.572 INFO:teuthology.orchestra.run.smithi051.stdout: debug auth = 20 2020-11-09T18:07:55.572 INFO:teuthology.orchestra.run.smithi051.stdout: mon reweight min pgs per osd = 4 2020-11-09T18:07:55.572 INFO:teuthology.orchestra.run.smithi051.stdout: mon reweight min bytes per osd = 10 2020-11-09T18:07:55.572 INFO:teuthology.orchestra.run.smithi051.stdout: mgr/telemetry/nag = false 2020-11-09T18:07:55.572 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:07:55.573 INFO:teuthology.orchestra.run.smithi051.stdout:[mon] 2020-11-09T18:07:55.573 INFO:teuthology.orchestra.run.smithi051.stdout: debug ms = 1 2020-11-09T18:07:55.573 INFO:teuthology.orchestra.run.smithi051.stdout: debug mon = 20 2020-11-09T18:07:55.573 INFO:teuthology.orchestra.run.smithi051.stdout: debug paxos = 20 2020-11-09T18:07:55.573 INFO:teuthology.orchestra.run.smithi051.stdout: debug auth = 20 2020-11-09T18:07:55.573 INFO:teuthology.orchestra.run.smithi051.stdout: mon data avail warn = 5 2020-11-09T18:07:55.574 INFO:teuthology.orchestra.run.smithi051.stdout: mon mgr mkfs grace = 240 2020-11-09T18:07:55.574 INFO:teuthology.orchestra.run.smithi051.stdout: mon reweight min pgs per osd = 4 2020-11-09T18:07:55.574 INFO:teuthology.orchestra.run.smithi051.stdout: mon osd reporter subtree level = osd 2020-11-09T18:07:55.574 INFO:teuthology.orchestra.run.smithi051.stdout: mon osd prime pg temp = true 2020-11-09T18:07:55.574 INFO:teuthology.orchestra.run.smithi051.stdout: mon reweight min bytes per osd = 10 2020-11-09T18:07:55.574 INFO:teuthology.orchestra.run.smithi051.stdout: mon osd crush smoke test = False 2020-11-09T18:07:55.574 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:07:55.575 INFO:teuthology.orchestra.run.smithi051.stdout:[client] 2020-11-09T18:07:55.575 INFO:teuthology.orchestra.run.smithi051.stdout: rgw cache enabled = true 2020-11-09T18:07:55.575 INFO:teuthology.orchestra.run.smithi051.stdout: rgw enable ops log = true 2020-11-09T18:07:55.575 INFO:teuthology.orchestra.run.smithi051.stdout: rgw enable usage log = true 2020-11-09T18:07:55.575 INFO:teuthology.orchestra.run.smithi051.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-09T18:07:55.575 INFO:teuthology.orchestra.run.smithi051.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-09T18:07:55.575 INFO:teuthology.orchestra.run.smithi051.stdout: debug rgw = 20 2020-11-09T18:07:55.576 INFO:teuthology.orchestra.run.smithi051.stdout: rgw crypt require ssl = False 2020-11-09T18:07:55.576 INFO:teuthology.orchestra.run.smithi051.stdout: rgw crypt s3 kms backend = testing 2020-11-09T18:07:55.576 INFO:teuthology.orchestra.run.smithi051.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-11-09T18:07:55.576 INFO:teuthology.orchestra.run.smithi051.stdout: rgw curl low speed time = 300 2020-11-09T18:07:55.576 INFO:teuthology.orchestra.run.smithi051.stdout: rgw data log num shards = 4 2020-11-09T18:07:55.576 INFO:teuthology.orchestra.run.smithi051.stdout: rgw md log max shards = 4 2020-11-09T18:07:55.576 INFO:teuthology.orchestra.run.smithi051.stdout: rgw sync log trim interval = 0 2020-11-09T18:07:55.577 INFO:teuthology.orchestra.run.smithi051.stdout: rgw sync obj etag verify = True 2020-11-09T18:07:55.577 INFO:teuthology.orchestra.run.smithi051.stdout: setgroup = ceph 2020-11-09T18:07:55.577 INFO:teuthology.orchestra.run.smithi051.stdout: setuser = ceph 2020-11-09T18:07:55.577 INFO:teuthology.orchestra.run.smithi051.stdout:[mon.a] 2020-11-09T18:07:55.577 INFO:teuthology.orchestra.run.smithi051.stdout:[osd.0] 2020-11-09T18:07:55.577 INFO:teuthology.orchestra.run.smithi051.stdout: osd_max_omap_entries_per_request = 10 2020-11-09T18:07:55.577 INFO:teuthology.orchestra.run.smithi051.stdout:[osd.1] 2020-11-09T18:07:55.577 INFO:teuthology.orchestra.run.smithi051.stdout: osd_max_omap_entries_per_request = 1000 2020-11-09T18:07:55.578 INFO:teuthology.orchestra.run.smithi051.stdout:[osd.2] 2020-11-09T18:07:55.578 INFO:teuthology.orchestra.run.smithi051.stdout: osd_max_omap_entries_per_request = 10000 2020-11-09T18:07:55.579 INFO:teuthology.orchestra.run.smithi078.stdout:[global] 2020-11-09T18:07:55.579 INFO:teuthology.orchestra.run.smithi078.stdout: chdir = "" 2020-11-09T18:07:55.579 INFO:teuthology.orchestra.run.smithi078.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-09T18:07:55.580 INFO:teuthology.orchestra.run.smithi078.stdout: auth supported = cephx 2020-11-09T18:07:55.580 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:07:55.580 INFO:teuthology.orchestra.run.smithi078.stdout: filestore xattr use omap = true 2020-11-09T18:07:55.580 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:07:55.580 INFO:teuthology.orchestra.run.smithi078.stdout: mon clock drift allowed = 1.000 2020-11-09T18:07:55.580 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:07:55.580 INFO:teuthology.orchestra.run.smithi078.stdout: osd crush chooseleaf type = 0 2020-11-09T18:07:55.581 INFO:teuthology.orchestra.run.smithi078.stdout: auth debug = true 2020-11-09T18:07:55.581 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:07:55.581 INFO:teuthology.orchestra.run.smithi078.stdout: ms die on old message = true 2020-11-09T18:07:55.581 INFO:teuthology.orchestra.run.smithi078.stdout: ms die on bug = true 2020-11-09T18:07:55.581 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:07:55.581 INFO:teuthology.orchestra.run.smithi078.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-09T18:07:55.581 INFO:teuthology.orchestra.run.smithi078.stdout: mon pg warn max object skew = 0 2020-11-09T18:07:55.582 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:07:55.582 INFO:teuthology.orchestra.run.smithi078.stdout: # disable pg_autoscaler by default for new pools 2020-11-09T18:07:55.582 INFO:teuthology.orchestra.run.smithi078.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-09T18:07:55.582 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:07:55.582 INFO:teuthology.orchestra.run.smithi078.stdout: osd pool default size = 2 2020-11-09T18:07:55.582 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:07:55.582 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd allow primary affinity = true 2020-11-09T18:07:55.582 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd allow pg remap = true 2020-11-09T18:07:55.583 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on legacy crush tunables = false 2020-11-09T18:07:55.583 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on crush straw calc version zero = false 2020-11-09T18:07:55.583 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on no sortbitwise = false 2020-11-09T18:07:55.583 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on osd down out interval zero = false 2020-11-09T18:07:55.583 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on too few osds = false 2020-11-09T18:07:55.583 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-09T18:07:55.583 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-09T18:07:55.584 INFO:teuthology.orchestra.run.smithi078.stdout: mon_allow_pool_size_one = true 2020-11-09T18:07:55.584 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:07:55.584 INFO:teuthology.orchestra.run.smithi078.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-09T18:07:55.584 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:07:55.584 INFO:teuthology.orchestra.run.smithi078.stdout: osd default data pool replay window = 5 2020-11-09T18:07:55.584 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:07:55.584 INFO:teuthology.orchestra.run.smithi078.stdout: mon allow pool delete = true 2020-11-09T18:07:55.585 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:07:55.585 INFO:teuthology.orchestra.run.smithi078.stdout: mon cluster log file level = debug 2020-11-09T18:07:55.585 INFO:teuthology.orchestra.run.smithi078.stdout: debug asserts on shutdown = true 2020-11-09T18:07:55.585 INFO:teuthology.orchestra.run.smithi078.stdout: mon host = 172.21.15.51 2020-11-09T18:07:55.585 INFO:teuthology.orchestra.run.smithi078.stdout: osd heartbeat grace = 40 2020-11-09T18:07:55.585 INFO:teuthology.orchestra.run.smithi078.stdout: fsid = f8cfda89-96d9-4149-8cfc-db3ea75def88 2020-11-09T18:07:55.585 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:07:55.586 INFO:teuthology.orchestra.run.smithi078.stdout:[osd] 2020-11-09T18:07:55.586 INFO:teuthology.orchestra.run.smithi078.stdout: osd journal size = 100 2020-11-09T18:07:55.586 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:07:55.586 INFO:teuthology.orchestra.run.smithi078.stdout: osd scrub load threshold = 5.0 2020-11-09T18:07:55.586 INFO:teuthology.orchestra.run.smithi078.stdout: osd scrub max interval = 600 2020-11-09T18:07:55.586 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:07:55.586 INFO:teuthology.orchestra.run.smithi078.stdout: osd recover clone overlap = true 2020-11-09T18:07:55.586 INFO:teuthology.orchestra.run.smithi078.stdout: osd recovery max chunk = 1048576 2020-11-09T18:07:55.587 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:07:55.587 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug shutdown = true 2020-11-09T18:07:55.587 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug op order = true 2020-11-09T18:07:55.587 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug verify stray on activate = true 2020-11-09T18:07:55.587 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:07:55.587 INFO:teuthology.orchestra.run.smithi078.stdout: osd open classes on start = true 2020-11-09T18:07:55.587 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug pg log writeout = true 2020-11-09T18:07:55.588 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:07:55.588 INFO:teuthology.orchestra.run.smithi078.stdout: osd deep scrub update digest min age = 30 2020-11-09T18:07:55.588 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:07:55.588 INFO:teuthology.orchestra.run.smithi078.stdout: osd map max advance = 10 2020-11-09T18:07:55.588 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:07:55.588 INFO:teuthology.orchestra.run.smithi078.stdout: journal zero on create = true 2020-11-09T18:07:55.588 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:07:55.588 INFO:teuthology.orchestra.run.smithi078.stdout: filestore ondisk finisher threads = 3 2020-11-09T18:07:55.589 INFO:teuthology.orchestra.run.smithi078.stdout: filestore apply finisher threads = 3 2020-11-09T18:07:55.589 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:07:55.589 INFO:teuthology.orchestra.run.smithi078.stdout: bdev debug aio = true 2020-11-09T18:07:55.589 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug misdirected ops = true 2020-11-09T18:07:55.589 INFO:teuthology.orchestra.run.smithi078.stdout: debug filestore = 20 2020-11-09T18:07:55.589 INFO:teuthology.orchestra.run.smithi078.stdout: debug journal = 20 2020-11-09T18:07:55.589 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 20 2020-11-09T18:07:55.590 INFO:teuthology.orchestra.run.smithi078.stdout: debug osd = 25 2020-11-09T18:07:55.590 INFO:teuthology.orchestra.run.smithi078.stdout: osd fast shutdown = False 2020-11-09T18:07:55.590 INFO:teuthology.orchestra.run.smithi078.stdout: rocksdb delete range threshold = 0 2020-11-09T18:07:55.590 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:07:55.590 INFO:teuthology.orchestra.run.smithi078.stdout:[mgr] 2020-11-09T18:07:55.590 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 1 2020-11-09T18:07:55.590 INFO:teuthology.orchestra.run.smithi078.stdout: debug mgr = 20 2020-11-09T18:07:55.591 INFO:teuthology.orchestra.run.smithi078.stdout: debug mon = 20 2020-11-09T18:07:55.591 INFO:teuthology.orchestra.run.smithi078.stdout: debug auth = 20 2020-11-09T18:07:55.591 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min pgs per osd = 4 2020-11-09T18:07:55.591 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min bytes per osd = 10 2020-11-09T18:07:55.591 INFO:teuthology.orchestra.run.smithi078.stdout: mgr/telemetry/nag = false 2020-11-09T18:07:55.591 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:07:55.591 INFO:teuthology.orchestra.run.smithi078.stdout:[mon] 2020-11-09T18:07:55.592 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 1 2020-11-09T18:07:55.592 INFO:teuthology.orchestra.run.smithi078.stdout: debug mon = 20 2020-11-09T18:07:55.592 INFO:teuthology.orchestra.run.smithi078.stdout: debug paxos = 20 2020-11-09T18:07:55.592 INFO:teuthology.orchestra.run.smithi078.stdout: debug auth = 20 2020-11-09T18:07:55.592 INFO:teuthology.orchestra.run.smithi078.stdout: mon data avail warn = 5 2020-11-09T18:07:55.592 INFO:teuthology.orchestra.run.smithi078.stdout: mon mgr mkfs grace = 240 2020-11-09T18:07:55.592 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min pgs per osd = 4 2020-11-09T18:07:55.592 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd reporter subtree level = osd 2020-11-09T18:07:55.593 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd prime pg temp = true 2020-11-09T18:07:55.593 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min bytes per osd = 10 2020-11-09T18:07:55.593 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd crush smoke test = False 2020-11-09T18:07:55.593 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:07:55.593 INFO:teuthology.orchestra.run.smithi078.stdout:[client] 2020-11-09T18:07:55.593 INFO:teuthology.orchestra.run.smithi078.stdout: rgw cache enabled = true 2020-11-09T18:07:55.593 INFO:teuthology.orchestra.run.smithi078.stdout: rgw enable ops log = true 2020-11-09T18:07:55.594 INFO:teuthology.orchestra.run.smithi078.stdout: rgw enable usage log = true 2020-11-09T18:07:55.594 INFO:teuthology.orchestra.run.smithi078.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-09T18:07:55.594 INFO:teuthology.orchestra.run.smithi078.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-09T18:07:55.594 INFO:teuthology.orchestra.run.smithi078.stdout: debug rgw = 20 2020-11-09T18:07:55.594 INFO:teuthology.orchestra.run.smithi078.stdout: rgw crypt require ssl = False 2020-11-09T18:07:55.594 INFO:teuthology.orchestra.run.smithi078.stdout: rgw crypt s3 kms backend = testing 2020-11-09T18:07:55.594 INFO:teuthology.orchestra.run.smithi078.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-11-09T18:07:55.594 INFO:teuthology.orchestra.run.smithi078.stdout: rgw curl low speed time = 300 2020-11-09T18:07:55.595 INFO:teuthology.orchestra.run.smithi078.stdout: rgw data log num shards = 4 2020-11-09T18:07:55.595 INFO:teuthology.orchestra.run.smithi078.stdout: rgw md log max shards = 4 2020-11-09T18:07:55.595 INFO:teuthology.orchestra.run.smithi078.stdout: rgw sync log trim interval = 0 2020-11-09T18:07:55.595 INFO:teuthology.orchestra.run.smithi078.stdout: rgw sync obj etag verify = True 2020-11-09T18:07:55.595 INFO:teuthology.orchestra.run.smithi078.stdout: setgroup = ceph 2020-11-09T18:07:55.595 INFO:teuthology.orchestra.run.smithi078.stdout: setuser = ceph 2020-11-09T18:07:55.595 INFO:teuthology.orchestra.run.smithi078.stdout:[mon.a] 2020-11-09T18:07:55.596 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.0] 2020-11-09T18:07:55.596 INFO:teuthology.orchestra.run.smithi078.stdout: osd_max_omap_entries_per_request = 10 2020-11-09T18:07:55.596 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.1] 2020-11-09T18:07:55.596 INFO:teuthology.orchestra.run.smithi078.stdout: osd_max_omap_entries_per_request = 1000 2020-11-09T18:07:55.596 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.2] 2020-11-09T18:07:55.596 INFO:teuthology.orchestra.run.smithi078.stdout: osd_max_omap_entries_per_request = 10000 2020-11-09T18:07:55.597 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2020-11-09T18:07:55.597 INFO:teuthology.orchestra.run.smithi051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2020-11-09T18:07:55.667 INFO:tasks.ceph:Copying monmap to all nodes... 2020-11-09T18:07:55.668 INFO:teuthology.orchestra.run.smithi051:> set -ex 2020-11-09T18:07:55.668 INFO:teuthology.orchestra.run.smithi051:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2020-11-09T18:07:55.673 INFO:teuthology.orchestra.run.smithi051.stderr:+ dd if=/etc/ceph/c1.keyring of=/dev/stdout 2020-11-09T18:07:55.674 INFO:teuthology.orchestra.run.smithi051.stdout:[mon.] 2020-11-09T18:07:55.674 INFO:teuthology.orchestra.run.smithi051.stdout: key = AQB7halfCKnQDhAAPT0jhH/mowifqecAqvjmEQ== 2020-11-09T18:07:55.674 INFO:teuthology.orchestra.run.smithi051.stdout:[client.admin] 2020-11-09T18:07:55.674 INFO:teuthology.orchestra.run.smithi051.stdout: key = AQB7halfbN/BJxAAAwk+P9yURXgEf2+IgeVptQ== 2020-11-09T18:07:55.674 INFO:teuthology.orchestra.run.smithi051.stdout: caps mds = "allow *" 2020-11-09T18:07:55.675 INFO:teuthology.orchestra.run.smithi051.stdout: caps mgr = "allow *" 2020-11-09T18:07:55.675 INFO:teuthology.orchestra.run.smithi051.stdout: caps mon = "allow *" 2020-11-09T18:07:55.675 INFO:teuthology.orchestra.run.smithi051.stdout: caps osd = "allow *" 2020-11-09T18:07:55.675 INFO:teuthology.orchestra.run.smithi051.stderr:0+1 records in 2020-11-09T18:07:55.675 INFO:teuthology.orchestra.run.smithi051.stderr:0+1 records out 2020-11-09T18:07:55.675 INFO:teuthology.orchestra.run.smithi051.stderr:206 bytes copied, 8.1027e-05 s, 2.5 MB/s 2020-11-09T18:07:55.676 INFO:teuthology.orchestra.run.smithi051:> set -ex 2020-11-09T18:07:55.676 INFO:teuthology.orchestra.run.smithi051:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2020-11-09T18:07:55.721 INFO:teuthology.orchestra.run.smithi051.stderr:+ dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2020-11-09T18:07:55.722 INFO:teuthology.orchestra.run.smithi051.stdout: ���ډ��AI���>�]�{��_�;�{��_�;��aXa �3��3a 2020-11-09T18:07:55.723 INFO:teuthology.orchestra.run.smithi051.stderr:0+1 records in 2020-11-09T18:07:55.723 INFO:teuthology.orchestra.run.smithi051.stderr:0+1 records out 2020-11-09T18:07:55.723 INFO:teuthology.orchestra.run.smithi051.stderr:201 bytes copied, 0.000158493 s, 1.3 MB/s 2020-11-09T18:07:55.723 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi051.front.sepia.ceph.com 2020-11-09T18:07:55.724 INFO:teuthology.orchestra.run.smithi051:> set -ex 2020-11-09T18:07:55.724 INFO:teuthology.orchestra.run.smithi051:> sudo dd of=/etc/ceph/c1.keyring 2020-11-09T18:07:55.724 INFO:teuthology.orchestra.run.smithi051:> sudo chmod 0644 /etc/ceph/c1.keyring 2020-11-09T18:07:55.769 INFO:teuthology.orchestra.run.smithi051.stderr:+ sudo dd of=/etc/ceph/c1.keyring 2020-11-09T18:07:55.779 INFO:teuthology.orchestra.run.smithi051.stderr:0+1 records in 2020-11-09T18:07:55.779 INFO:teuthology.orchestra.run.smithi051.stderr:0+1 records out 2020-11-09T18:07:55.779 INFO:teuthology.orchestra.run.smithi051.stderr:206 bytes copied, 0.00013166 s, 1.6 MB/s 2020-11-09T18:07:55.780 INFO:teuthology.orchestra.run.smithi051.stderr:+ sudo chmod 0644 /etc/ceph/c1.keyring 2020-11-09T18:07:55.789 INFO:teuthology.orchestra.run.smithi051:> set -ex 2020-11-09T18:07:55.790 INFO:teuthology.orchestra.run.smithi051:> dd of=/home/ubuntu/cephtest/c1.monmap 2020-11-09T18:07:55.837 INFO:teuthology.orchestra.run.smithi051.stderr:+ dd of=/home/ubuntu/cephtest/c1.monmap 2020-11-09T18:07:55.839 INFO:teuthology.orchestra.run.smithi051.stderr:0+1 records in 2020-11-09T18:07:55.839 INFO:teuthology.orchestra.run.smithi051.stderr:0+1 records out 2020-11-09T18:07:55.839 INFO:teuthology.orchestra.run.smithi051.stderr:201 bytes copied, 0.00012748 s, 1.6 MB/s 2020-11-09T18:07:55.839 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi078.front.sepia.ceph.com 2020-11-09T18:07:55.840 INFO:teuthology.orchestra.run.smithi078:> set -ex 2020-11-09T18:07:55.840 INFO:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/ceph/c1.keyring 2020-11-09T18:07:55.840 INFO:teuthology.orchestra.run.smithi078:> sudo chmod 0644 /etc/ceph/c1.keyring 2020-11-09T18:07:55.845 INFO:teuthology.orchestra.run.smithi078.stderr:+ sudo dd of=/etc/ceph/c1.keyring 2020-11-09T18:07:55.854 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records in 2020-11-09T18:07:55.854 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records out 2020-11-09T18:07:55.855 INFO:teuthology.orchestra.run.smithi078.stderr:206 bytes copied, 6.736e-05 s, 3.1 MB/s 2020-11-09T18:07:55.855 INFO:teuthology.orchestra.run.smithi078.stderr:+ sudo chmod 0644 /etc/ceph/c1.keyring 2020-11-09T18:07:55.865 INFO:teuthology.orchestra.run.smithi078:> set -ex 2020-11-09T18:07:55.866 INFO:teuthology.orchestra.run.smithi078:> dd of=/home/ubuntu/cephtest/c1.monmap 2020-11-09T18:07:55.914 INFO:teuthology.orchestra.run.smithi078.stderr:+ dd of=/home/ubuntu/cephtest/c1.monmap 2020-11-09T18:07:55.915 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records in 2020-11-09T18:07:55.916 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records out 2020-11-09T18:07:55.916 INFO:teuthology.orchestra.run.smithi078.stderr:201 bytes copied, 7.4784e-05 s, 2.7 MB/s 2020-11-09T18:07:55.916 INFO:tasks.ceph:Setting up mon nodes... 2020-11-09T18:07:55.916 INFO:tasks.ceph:Setting up mgr nodes... 2020-11-09T18:07:55.917 INFO:teuthology.orchestra.run.smithi051:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2020-11-09T18:07:55.960 INFO:teuthology.orchestra.run.smithi051.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2020-11-09T18:07:55.962 INFO:tasks.ceph:Setting up mds nodes... 2020-11-09T18:07:55.962 INFO:tasks.ceph_client:Setting up client nodes... 2020-11-09T18:07:55.963 INFO:teuthology.orchestra.run.smithi051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2020-11-09T18:07:56.039 INFO:teuthology.orchestra.run.smithi051.stdout:creating /etc/ceph/c1.client.0.keyring 2020-11-09T18:07:56.050 INFO:teuthology.orchestra.run.smithi051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2020-11-09T18:07:56.124 INFO:teuthology.orchestra.run.smithi051.stdout:creating /etc/ceph/c1.client.1.keyring 2020-11-09T18:07:56.136 INFO:tasks.ceph:Running mkfs on osd nodes... 2020-11-09T18:07:56.136 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi051.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2020-11-09T18:07:56.136 INFO:teuthology.orchestra.run.smithi051:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2020-11-09T18:07:56.191 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-09T18:07:56.191 INFO:tasks.ceph:role: c1.osd.0 2020-11-09T18:07:56.192 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi051.front.sepia.ceph.com 2020-11-09T18:07:56.192 INFO:teuthology.orchestra.run.smithi051:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-11-09T18:07:56.721 INFO:teuthology.orchestra.run.smithi051.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-11-09T18:07:56.722 INFO:teuthology.orchestra.run.smithi051.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-09T18:07:56.722 INFO:teuthology.orchestra.run.smithi051.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-11-09T18:07:56.722 INFO:teuthology.orchestra.run.smithi051.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-09T18:07:56.722 INFO:teuthology.orchestra.run.smithi051.stdout: = sunit=0 swidth=0 blks 2020-11-09T18:07:56.723 INFO:teuthology.orchestra.run.smithi051.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-11-09T18:07:56.723 INFO:teuthology.orchestra.run.smithi051.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-09T18:07:56.723 INFO:teuthology.orchestra.run.smithi051.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-09T18:07:56.723 INFO:teuthology.orchestra.run.smithi051.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-09T18:07:56.724 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi051.front.sepia.ceph.com -o noatime 2020-11-09T18:07:56.724 INFO:teuthology.orchestra.run.smithi051:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2020-11-09T18:07:56.785 INFO:teuthology.orchestra.run.smithi051:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2020-11-09T18:07:56.797 INFO:teuthology.orchestra.run.smithi051.stderr:sudo: /sbin/restorecon: command not found 2020-11-09T18:07:56.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-09T18:07:56.799 INFO:teuthology.orchestra.run.smithi051:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2020-11-09T18:07:56.854 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-09T18:07:56.855 INFO:tasks.ceph:role: c1.osd.1 2020-11-09T18:07:56.855 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi051.front.sepia.ceph.com 2020-11-09T18:07:56.855 INFO:teuthology.orchestra.run.smithi051:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-11-09T18:07:57.337 INFO:teuthology.orchestra.run.smithi051.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-11-09T18:07:57.338 INFO:teuthology.orchestra.run.smithi051.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-09T18:07:57.338 INFO:teuthology.orchestra.run.smithi051.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-11-09T18:07:57.338 INFO:teuthology.orchestra.run.smithi051.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-09T18:07:57.339 INFO:teuthology.orchestra.run.smithi051.stdout: = sunit=0 swidth=0 blks 2020-11-09T18:07:57.339 INFO:teuthology.orchestra.run.smithi051.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-11-09T18:07:57.339 INFO:teuthology.orchestra.run.smithi051.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-09T18:07:57.339 INFO:teuthology.orchestra.run.smithi051.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-09T18:07:57.339 INFO:teuthology.orchestra.run.smithi051.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-09T18:07:57.340 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi051.front.sepia.ceph.com -o noatime 2020-11-09T18:07:57.340 INFO:teuthology.orchestra.run.smithi051:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2020-11-09T18:07:57.374 INFO:teuthology.orchestra.run.smithi051:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2020-11-09T18:07:57.427 INFO:teuthology.orchestra.run.smithi051.stderr:sudo: /sbin/restorecon: command not found 2020-11-09T18:07:57.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-09T18:07:57.429 INFO:teuthology.orchestra.run.smithi051:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2020-11-09T18:07:57.483 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-09T18:07:57.484 INFO:tasks.ceph:role: c1.osd.2 2020-11-09T18:07:57.484 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi051.front.sepia.ceph.com 2020-11-09T18:07:57.484 INFO:teuthology.orchestra.run.smithi051:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2020-11-09T18:07:58.143 INFO:teuthology.orchestra.run.smithi051.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2020-11-09T18:07:58.144 INFO:teuthology.orchestra.run.smithi051.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-09T18:07:58.144 INFO:teuthology.orchestra.run.smithi051.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-11-09T18:07:58.144 INFO:teuthology.orchestra.run.smithi051.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-09T18:07:58.144 INFO:teuthology.orchestra.run.smithi051.stdout: = sunit=0 swidth=0 blks 2020-11-09T18:07:58.145 INFO:teuthology.orchestra.run.smithi051.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-11-09T18:07:58.145 INFO:teuthology.orchestra.run.smithi051.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-09T18:07:58.145 INFO:teuthology.orchestra.run.smithi051.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-09T18:07:58.145 INFO:teuthology.orchestra.run.smithi051.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-09T18:07:58.146 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi051.front.sepia.ceph.com -o noatime 2020-11-09T18:07:58.146 INFO:teuthology.orchestra.run.smithi051:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2020-11-09T18:07:58.217 INFO:teuthology.orchestra.run.smithi051:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2020-11-09T18:07:58.229 INFO:teuthology.orchestra.run.smithi051.stderr:sudo: /sbin/restorecon: command not found 2020-11-09T18:07:58.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-09T18:07:58.230 INFO:teuthology.orchestra.run.smithi051:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2020-11-09T18:07:58.328 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:07:58.325+0000 7f491506cd80 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2020-11-09T18:07:58.333 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:07:58.333+0000 7f491506cd80 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2020-11-09T18:07:58.334 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:07:58.333+0000 7f491506cd80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2020-11-09T18:07:58.334 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:07:58.333+0000 7f491506cd80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2020-11-09T18:07:58.334 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:07:58.333+0000 7f491506cd80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2020-11-09T18:07:58.339 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:07:58.337+0000 7f491506cd80 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2020-11-09T18:08:00.536 INFO:teuthology.orchestra.run.smithi051:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2020-11-09T18:08:00.551 INFO:teuthology.orchestra.run.smithi051:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2020-11-09T18:08:00.633 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:08:00.629+0000 7fa8abe8ed80 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2020-11-09T18:08:00.633 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:08:00.629+0000 7fa8abe8ed80 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2020-11-09T18:08:00.633 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:08:00.629+0000 7fa8abe8ed80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2020-11-09T18:08:00.634 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:08:00.629+0000 7fa8abe8ed80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2020-11-09T18:08:00.634 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:08:00.629+0000 7fa8abe8ed80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2020-11-09T18:08:00.635 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:08:00.633+0000 7fa8abe8ed80 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2020-11-09T18:08:02.809 INFO:teuthology.orchestra.run.smithi051:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2020-11-09T18:08:02.868 INFO:teuthology.orchestra.run.smithi051:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2020-11-09T18:08:02.954 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:08:02.953+0000 7fe54319dd80 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2020-11-09T18:08:02.954 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:08:02.953+0000 7fe54319dd80 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2020-11-09T18:08:02.954 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:08:02.953+0000 7fe54319dd80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2020-11-09T18:08:02.955 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:08:02.953+0000 7fe54319dd80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2020-11-09T18:08:02.955 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:08:02.953+0000 7fe54319dd80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2020-11-09T18:08:02.955 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:08:02.953+0000 7fe54319dd80 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2020-11-09T18:08:05.128 INFO:teuthology.orchestra.run.smithi051:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2020-11-09T18:08:05.184 INFO:tasks.ceph:Reading keys from all nodes... 2020-11-09T18:08:05.185 INFO:teuthology.orchestra.run.smithi051:> set -ex 2020-11-09T18:08:05.185 INFO:teuthology.orchestra.run.smithi051:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2020-11-09T18:08:05.229 INFO:teuthology.orchestra.run.smithi051.stderr:+ sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2020-11-09T18:08:05.238 INFO:teuthology.orchestra.run.smithi051.stdout:[mgr.x] 2020-11-09T18:08:05.238 INFO:teuthology.orchestra.run.smithi051.stdout: key = AQB7halfYWtXORAADUAE8cs41FkSrEAirXVmCQ== 2020-11-09T18:08:05.239 INFO:teuthology.orchestra.run.smithi051.stderr:0+1 records in 2020-11-09T18:08:05.239 INFO:teuthology.orchestra.run.smithi051.stderr:0+1 records out 2020-11-09T18:08:05.239 INFO:teuthology.orchestra.run.smithi051.stderr:56 bytes copied, 0.00017368 s, 322 kB/s 2020-11-09T18:08:05.240 INFO:teuthology.orchestra.run.smithi051:> set -ex 2020-11-09T18:08:05.241 INFO:teuthology.orchestra.run.smithi051:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2020-11-09T18:08:05.285 INFO:teuthology.orchestra.run.smithi051.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2020-11-09T18:08:05.294 INFO:teuthology.orchestra.run.smithi051.stdout:[osd.0] 2020-11-09T18:08:05.294 INFO:teuthology.orchestra.run.smithi051.stdout: key = AQB+halfMn+vExAAvHMdb9VM//SP9a7WQIQVbg== 2020-11-09T18:08:05.295 INFO:teuthology.orchestra.run.smithi051.stderr:0+1 records in 2020-11-09T18:08:05.295 INFO:teuthology.orchestra.run.smithi051.stderr:0+1 records out 2020-11-09T18:08:05.295 INFO:teuthology.orchestra.run.smithi051.stderr:56 bytes copied, 0.00016889 s, 332 kB/s 2020-11-09T18:08:05.296 INFO:teuthology.orchestra.run.smithi051:> set -ex 2020-11-09T18:08:05.296 INFO:teuthology.orchestra.run.smithi051:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2020-11-09T18:08:05.341 INFO:teuthology.orchestra.run.smithi051.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2020-11-09T18:08:05.350 INFO:teuthology.orchestra.run.smithi051.stdout:[osd.1] 2020-11-09T18:08:05.350 INFO:teuthology.orchestra.run.smithi051.stdout: key = AQCAhalfUmjOJRAAU22T5vBsWbkkhCgnanqdgA== 2020-11-09T18:08:05.351 INFO:teuthology.orchestra.run.smithi051.stderr:0+1 records in 2020-11-09T18:08:05.351 INFO:teuthology.orchestra.run.smithi051.stderr:0+1 records out 2020-11-09T18:08:05.351 INFO:teuthology.orchestra.run.smithi051.stderr:56 bytes copied, 0.00017734 s, 316 kB/s 2020-11-09T18:08:05.352 INFO:teuthology.orchestra.run.smithi051:> set -ex 2020-11-09T18:08:05.353 INFO:teuthology.orchestra.run.smithi051:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2020-11-09T18:08:05.397 INFO:teuthology.orchestra.run.smithi051.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2020-11-09T18:08:05.406 INFO:teuthology.orchestra.run.smithi051.stdout:[osd.2] 2020-11-09T18:08:05.406 INFO:teuthology.orchestra.run.smithi051.stdout: key = AQCChalfbYP7OBAAZ6gAYji5/SSTAmysmoimPQ== 2020-11-09T18:08:05.406 INFO:teuthology.orchestra.run.smithi051.stderr:0+1 records in 2020-11-09T18:08:05.406 INFO:teuthology.orchestra.run.smithi051.stderr:0+1 records out 2020-11-09T18:08:05.406 INFO:teuthology.orchestra.run.smithi051.stderr:56 bytes copied, 0.000161167 s, 347 kB/s 2020-11-09T18:08:05.407 INFO:teuthology.orchestra.run.smithi051:> set -ex 2020-11-09T18:08:05.408 INFO:teuthology.orchestra.run.smithi051:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2020-11-09T18:08:05.453 INFO:teuthology.orchestra.run.smithi051.stderr:+ dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2020-11-09T18:08:05.454 INFO:teuthology.orchestra.run.smithi051.stdout:[client.0] 2020-11-09T18:08:05.454 INFO:teuthology.orchestra.run.smithi051.stdout: key = AQB8half7LVpAhAA30XMs03/MnYjh1Z6pZyiWQ== 2020-11-09T18:08:05.455 INFO:teuthology.orchestra.run.smithi051.stderr:0+1 records in 2020-11-09T18:08:05.455 INFO:teuthology.orchestra.run.smithi051.stderr:0+1 records out 2020-11-09T18:08:05.455 INFO:teuthology.orchestra.run.smithi051.stderr:59 bytes copied, 0.000164723 s, 358 kB/s 2020-11-09T18:08:05.455 INFO:teuthology.orchestra.run.smithi051:> set -ex 2020-11-09T18:08:05.455 INFO:teuthology.orchestra.run.smithi051:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2020-11-09T18:08:05.501 INFO:teuthology.orchestra.run.smithi051.stderr:+ dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2020-11-09T18:08:05.502 INFO:teuthology.orchestra.run.smithi051.stdout:[client.1] 2020-11-09T18:08:05.502 INFO:teuthology.orchestra.run.smithi051.stdout: key = AQB8halfKPmCBxAAIgbyUR11YnXb5J3ofZUXNw== 2020-11-09T18:08:05.503 INFO:teuthology.orchestra.run.smithi051.stderr:0+1 records in 2020-11-09T18:08:05.503 INFO:teuthology.orchestra.run.smithi051.stderr:0+1 records out 2020-11-09T18:08:05.503 INFO:teuthology.orchestra.run.smithi051.stderr:59 bytes copied, 0.000156707 s, 376 kB/s 2020-11-09T18:08:05.503 INFO:tasks.ceph:Adding keys to all mons... 2020-11-09T18:08:05.503 INFO:teuthology.orchestra.run.smithi051:> sudo tee -a /etc/ceph/c1.keyring 2020-11-09T18:08:05.560 INFO:teuthology.orchestra.run.smithi051.stdout:[mgr.x] 2020-11-09T18:08:05.560 INFO:teuthology.orchestra.run.smithi051.stdout: key = AQB7halfYWtXORAADUAE8cs41FkSrEAirXVmCQ== 2020-11-09T18:08:05.561 INFO:teuthology.orchestra.run.smithi051.stdout:[osd.0] 2020-11-09T18:08:05.561 INFO:teuthology.orchestra.run.smithi051.stdout: key = AQB+halfMn+vExAAvHMdb9VM//SP9a7WQIQVbg== 2020-11-09T18:08:05.561 INFO:teuthology.orchestra.run.smithi051.stdout:[osd.1] 2020-11-09T18:08:05.561 INFO:teuthology.orchestra.run.smithi051.stdout: key = AQCAhalfUmjOJRAAU22T5vBsWbkkhCgnanqdgA== 2020-11-09T18:08:05.562 INFO:teuthology.orchestra.run.smithi051.stdout:[osd.2] 2020-11-09T18:08:05.562 INFO:teuthology.orchestra.run.smithi051.stdout: key = AQCChalfbYP7OBAAZ6gAYji5/SSTAmysmoimPQ== 2020-11-09T18:08:05.562 INFO:teuthology.orchestra.run.smithi051.stdout:[client.0] 2020-11-09T18:08:05.563 INFO:teuthology.orchestra.run.smithi051.stdout: key = AQB8half7LVpAhAA30XMs03/MnYjh1Z6pZyiWQ== 2020-11-09T18:08:05.568 INFO:teuthology.orchestra.run.smithi051.stdout:[client.1] 2020-11-09T18:08:05.568 INFO:teuthology.orchestra.run.smithi051.stdout: key = AQB8halfKPmCBxAAIgbyUR11YnXb5J3ofZUXNw== 2020-11-09T18:08:05.569 INFO:teuthology.orchestra.run.smithi051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2020-11-09T18:08:05.644 INFO:teuthology.orchestra.run.smithi051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-09T18:08:05.727 INFO:teuthology.orchestra.run.smithi051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-09T18:08:05.811 INFO:teuthology.orchestra.run.smithi051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-09T18:08:05.895 INFO:teuthology.orchestra.run.smithi051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-11-09T18:08:05.982 INFO:teuthology.orchestra.run.smithi051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-11-09T18:08:06.021 INFO:tasks.ceph:Running mkfs on mon nodes... 2020-11-09T18:08:06.022 INFO:teuthology.orchestra.run.smithi051:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2020-11-09T18:08:06.080 INFO:teuthology.orchestra.run.smithi051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2020-11-09T18:08:06.177 INFO:teuthology.orchestra.run.smithi051:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2020-11-09T18:08:06.191 INFO:teuthology.orchestra.run.smithi051:> rm -- /home/ubuntu/cephtest/c1.monmap 2020-11-09T18:08:06.239 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2020-11-09T18:08:06.240 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2020-11-09T18:08:06.240 INFO:teuthology.orchestra.run.smithi051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2020-11-09T18:08:06.282 INFO:tasks.ceph.c1.mon.a:Started 2020-11-09T18:08:06.283 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2020-11-09T18:08:06.283 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2020-11-09T18:08:06.283 INFO:teuthology.orchestra.run.smithi051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2020-11-09T18:08:06.286 INFO:tasks.ceph.c1.mgr.x:Started 2020-11-09T18:08:06.286 INFO:tasks.ceph:Setting crush tunables to default 2020-11-09T18:08:06.286 INFO:teuthology.orchestra.run.smithi051:> sudo ceph --cluster c1 osd crush tunables default 2020-11-09T18:08:06.640 INFO:teuthology.orchestra.run.smithi051.stderr:adjusted tunables profile to default 2020-11-09T18:08:06.653 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2020-11-09T18:08:06.654 INFO:teuthology.orchestra.run.smithi051:> set -ex 2020-11-09T18:08:06.655 INFO:teuthology.orchestra.run.smithi051:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2020-11-09T18:08:06.661 INFO:teuthology.orchestra.run.smithi051.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2020-11-09T18:08:06.670 INFO:teuthology.orchestra.run.smithi051.stdout:69041991-91cc-444e-a8c0-1444e1d4b5bd 2020-11-09T18:08:06.670 INFO:teuthology.orchestra.run.smithi051.stderr:0+1 records in 2020-11-09T18:08:06.671 INFO:teuthology.orchestra.run.smithi051.stderr:0+1 records out 2020-11-09T18:08:06.671 INFO:teuthology.orchestra.run.smithi051.stderr:37 bytes copied, 8.6793e-05 s, 426 kB/s 2020-11-09T18:08:06.672 INFO:teuthology.orchestra.run.smithi051:> set -ex 2020-11-09T18:08:06.673 INFO:teuthology.orchestra.run.smithi051:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2020-11-09T18:08:06.716 INFO:teuthology.orchestra.run.smithi051.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2020-11-09T18:08:06.727 INFO:teuthology.orchestra.run.smithi051.stdout:24120236-930c-4195-adc7-d15fe3d0030d 2020-11-09T18:08:06.727 INFO:teuthology.orchestra.run.smithi051.stderr:0+1 records in 2020-11-09T18:08:06.728 INFO:teuthology.orchestra.run.smithi051.stderr:0+1 records out 2020-11-09T18:08:06.728 INFO:teuthology.orchestra.run.smithi051.stderr:37 bytes copied, 0.000146248 s, 253 kB/s 2020-11-09T18:08:06.730 INFO:teuthology.orchestra.run.smithi051:> set -ex 2020-11-09T18:08:06.730 INFO:teuthology.orchestra.run.smithi051:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2020-11-09T18:08:06.778 INFO:teuthology.orchestra.run.smithi051.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2020-11-09T18:08:06.789 INFO:teuthology.orchestra.run.smithi051.stdout:9f230dcb-16c5-4b08-a5ed-bb5ef1608640 2020-11-09T18:08:06.789 INFO:teuthology.orchestra.run.smithi051.stderr:0+1 records in 2020-11-09T18:08:06.789 INFO:teuthology.orchestra.run.smithi051.stderr:0+1 records out 2020-11-09T18:08:06.790 INFO:teuthology.orchestra.run.smithi051.stderr:37 bytes copied, 0.000173861 s, 213 kB/s 2020-11-09T18:08:06.790 INFO:teuthology.orchestra.run.smithi051:> sudo ceph --cluster c1 osd new 69041991-91cc-444e-a8c0-1444e1d4b5bd 0 2020-11-09T18:08:07.059 INFO:teuthology.orchestra.run.smithi051.stdout:0 2020-11-09T18:08:07.074 INFO:teuthology.orchestra.run.smithi051:> sudo ceph --cluster c1 osd new 24120236-930c-4195-adc7-d15fe3d0030d 1 2020-11-09T18:08:07.292 INFO:teuthology.orchestra.run.smithi051.stdout:1 2020-11-09T18:08:07.305 INFO:teuthology.orchestra.run.smithi051:> sudo ceph --cluster c1 osd new 9f230dcb-16c5-4b08-a5ed-bb5ef1608640 2 2020-11-09T18:08:07.525 INFO:teuthology.orchestra.run.smithi051.stdout:2 2020-11-09T18:08:07.538 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2020-11-09T18:08:07.538 INFO:teuthology.orchestra.run.smithi051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2020-11-09T18:08:07.540 INFO:tasks.ceph.c1.osd.0:Started 2020-11-09T18:08:07.541 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2020-11-09T18:08:07.541 INFO:teuthology.orchestra.run.smithi051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2020-11-09T18:08:07.543 INFO:tasks.ceph.c1.osd.1:Started 2020-11-09T18:08:07.543 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2020-11-09T18:08:07.544 INFO:teuthology.orchestra.run.smithi051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2020-11-09T18:08:07.547 INFO:tasks.ceph.c1.osd.2:Started 2020-11-09T18:08:07.547 INFO:tasks.ceph:Waiting for OSDs to come up 2020-11-09T18:08:07.850 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2020-11-09T18:08:08.043 INFO:teuthology.misc.health.smithi051.stdout: 2020-11-09T18:08:08.043 INFO:teuthology.misc.health.smithi051.stdout:{"epoch":5,"fsid":"f8cfda89-96d9-4149-8cfc-db3ea75def88","created":"2020-11-09T18:08:06.436693+0000","modified":"2020-11-09T18:08:07.523940+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"69041991-91cc-444e-a8c0-1444e1d4b5bd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24120236-930c-4195-adc7-d15fe3d0030d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9f230dcb-16c5-4b08-a5ed-bb5ef1608640","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-09T18:08:08.054 DEBUG:teuthology.misc:0 of 3 OSDs are up 2020-11-09T18:08:08.159 INFO:tasks.ceph.c1.osd.1.smithi051.stderr:2020-11-09T18:08:08.157+0000 7ff7600ded80 -1 Falling back to public interface 2020-11-09T18:08:08.159 INFO:tasks.ceph.c1.osd.2.smithi051.stderr:2020-11-09T18:08:08.157+0000 7f4048cc5d80 -1 Falling back to public interface 2020-11-09T18:08:08.163 INFO:tasks.ceph.c1.osd.0.smithi051.stderr:2020-11-09T18:08:08.161+0000 7f21c2bdcd80 -1 Falling back to public interface 2020-11-09T18:08:08.552 INFO:tasks.ceph.c1.osd.2.smithi051.stderr:2020-11-09T18:08:08.549+0000 7f4048cc5d80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-09T18:08:08.770 INFO:tasks.ceph.c1.osd.0.smithi051.stderr:2020-11-09T18:08:08.769+0000 7f21c2bdcd80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-09T18:08:08.774 INFO:tasks.ceph.c1.osd.1.smithi051.stderr:2020-11-09T18:08:08.773+0000 7ff7600ded80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-09T18:08:10.467 INFO:tasks.ceph.c1.osd.0.smithi051.stderr:2020-11-09T18:08:10.465+0000 7f21ba0a6700 -1 osd.0 0 waiting for initial osdmap 2020-11-09T18:08:10.468 INFO:tasks.ceph.c1.osd.2.smithi051.stderr:2020-11-09T18:08:10.465+0000 7f404018f700 -1 osd.2 0 waiting for initial osdmap 2020-11-09T18:08:10.468 INFO:tasks.ceph.c1.osd.1.smithi051.stderr:2020-11-09T18:08:10.465+0000 7ff758dab700 -1 osd.1 0 waiting for initial osdmap 2020-11-09T18:08:14.358 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2020-11-09T18:08:14.651 INFO:teuthology.misc.health.smithi051.stdout: 2020-11-09T18:08:14.652 INFO:teuthology.misc.health.smithi051.stdout:{"epoch":10,"fsid":"f8cfda89-96d9-4149-8cfc-db3ea75def88","created":"2020-11-09T18:08:06.436693+0000","modified":"2020-11-09T18:08:13.486388+0000","last_up_change":"2020-11-09T18:08:11.469550+0000","last_in_change":"2020-11-09T18:08:11.469550+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-09T18:08:10.602093+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"69041991-91cc-444e-a8c0-1444e1d4b5bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6816","nonce":14202},{"type":"v1","addr":"172.21.15.51:6817","nonce":14202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6818","nonce":14202},{"type":"v1","addr":"172.21.15.51:6819","nonce":14202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6822","nonce":14202},{"type":"v1","addr":"172.21.15.51:6823","nonce":14202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6820","nonce":14202},{"type":"v1","addr":"172.21.15.51:6821","nonce":14202}]},"public_addr":"172.21.15.51:6817/14202","cluster_addr":"172.21.15.51:6819/14202","heartbeat_back_addr":"172.21.15.51:6823/14202","heartbeat_front_addr":"172.21.15.51:6821/14202","state":["exists","up"]},{"osd":1,"uuid":"24120236-930c-4195-adc7-d15fe3d0030d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":8,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6800","nonce":14199},{"type":"v1","addr":"172.21.15.51:6801","nonce":14199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6802","nonce":14199},{"type":"v1","addr":"172.21.15.51:6804","nonce":14199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6814","nonce":14199},{"type":"v1","addr":"172.21.15.51:6815","nonce":14199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6807","nonce":14199},{"type":"v1","addr":"172.21.15.51:6810","nonce":14199}]},"public_addr":"172.21.15.51:6801/14199","cluster_addr":"172.21.15.51:6804/14199","heartbeat_back_addr":"172.21.15.51:6815/14199","heartbeat_front_addr":"172.21.15.51:6810/14199","state":["exists","up"]},{"osd":2,"uuid":"9f230dcb-16c5-4b08-a5ed-bb5ef1608640","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6803","nonce":14200},{"type":"v1","addr":"172.21.15.51:6805","nonce":14200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6806","nonce":14200},{"type":"v1","addr":"172.21.15.51:6808","nonce":14200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6812","nonce":14200},{"type":"v1","addr":"172.21.15.51:6813","nonce":14200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6809","nonce":14200},{"type":"v1","addr":"172.21.15.51:6811","nonce":14200}]},"public_addr":"172.21.15.51:6805/14200","cluster_addr":"172.21.15.51:6808/14200","heartbeat_back_addr":"172.21.15.51:6813/14200","heartbeat_front_addr":"172.21.15.51:6811/14200","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T18:08:09.769540+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T18:08:09.758329+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T18:08:09.523430+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-09T18:08:14.665 DEBUG:teuthology.misc:3 of 3 OSDs are up 2020-11-09T18:08:14.665 INFO:tasks.ceph:Creating RBD pool 2020-11-09T18:08:14.666 INFO:teuthology.orchestra.run.smithi051:> sudo ceph --cluster c1 osd pool create rbd 8 2020-11-09T18:08:15.516 INFO:teuthology.orchestra.run.smithi051.stderr:pool 'rbd' created 2020-11-09T18:08:15.529 INFO:teuthology.orchestra.run.smithi051:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2020-11-09T18:08:16.521 INFO:teuthology.orchestra.run.smithi051.stderr:enabled application 'rbd' on pool 'rbd' 2020-11-09T18:08:16.535 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2020-11-09T18:08:16.536 INFO:teuthology.orchestra.run.smithi051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2020-11-09T18:08:16.536 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2020-11-09T18:08:16.829 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:08:16.830 INFO:teuthology.orchestra.run.smithi051.stdout:{"epoch":12,"fsid":"f8cfda89-96d9-4149-8cfc-db3ea75def88","created":"2020-11-09T18:08:06.436693+0000","modified":"2020-11-09T18:08:16.503030+0000","last_up_change":"2020-11-09T18:08:11.469550+0000","last_in_change":"2020-11-09T18:08:11.469550+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-09T18:08:10.602093+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-09T18:08:14.965591+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"69041991-91cc-444e-a8c0-1444e1d4b5bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6816","nonce":14202},{"type":"v1","addr":"172.21.15.51:6817","nonce":14202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6818","nonce":14202},{"type":"v1","addr":"172.21.15.51:6819","nonce":14202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6822","nonce":14202},{"type":"v1","addr":"172.21.15.51:6823","nonce":14202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6820","nonce":14202},{"type":"v1","addr":"172.21.15.51:6821","nonce":14202}]},"public_addr":"172.21.15.51:6817/14202","cluster_addr":"172.21.15.51:6819/14202","heartbeat_back_addr":"172.21.15.51:6823/14202","heartbeat_front_addr":"172.21.15.51:6821/14202","state":["exists","up"]},{"osd":1,"uuid":"24120236-930c-4195-adc7-d15fe3d0030d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6800","nonce":14199},{"type":"v1","addr":"172.21.15.51:6801","nonce":14199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6802","nonce":14199},{"type":"v1","addr":"172.21.15.51:6804","nonce":14199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6814","nonce":14199},{"type":"v1","addr":"172.21.15.51:6815","nonce":14199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6807","nonce":14199},{"type":"v1","addr":"172.21.15.51:6810","nonce":14199}]},"public_addr":"172.21.15.51:6801/14199","cluster_addr":"172.21.15.51:6804/14199","heartbeat_back_addr":"172.21.15.51:6815/14199","heartbeat_front_addr":"172.21.15.51:6810/14199","state":["exists","up"]},{"osd":2,"uuid":"9f230dcb-16c5-4b08-a5ed-bb5ef1608640","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6803","nonce":14200},{"type":"v1","addr":"172.21.15.51:6805","nonce":14200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6806","nonce":14200},{"type":"v1","addr":"172.21.15.51:6808","nonce":14200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6812","nonce":14200},{"type":"v1","addr":"172.21.15.51:6813","nonce":14200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6809","nonce":14200},{"type":"v1","addr":"172.21.15.51:6811","nonce":14200}]},"public_addr":"172.21.15.51:6805/14200","cluster_addr":"172.21.15.51:6808/14200","heartbeat_back_addr":"172.21.15.51:6813/14200","heartbeat_front_addr":"172.21.15.51:6811/14200","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T18:08:09.769540+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T18:08:09.758329+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T18:08:09.523430+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-09T18:08:16.844 INFO:tasks.ceph.ceph_manager.c1:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-09T18:08:10.602093+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '10', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-09T18:08:14.965591+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-11-09T18:08:16.845 INFO:teuthology.orchestra.run.smithi051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd pool get device_health_metrics pg_num 2020-11-09T18:08:17.137 INFO:teuthology.orchestra.run.smithi051.stdout:pg_num: 1 2020-11-09T18:08:17.152 INFO:teuthology.orchestra.run.smithi051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd pool get rbd pg_num 2020-11-09T18:08:17.448 INFO:teuthology.orchestra.run.smithi051.stdout:pg_num: 8 2020-11-09T18:08:17.463 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2020-11-09T18:08:17.463 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2020-11-09T18:08:17.463 INFO:teuthology.orchestra.run.smithi051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2020-11-09T18:08:17.790 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:08:17.805 INFO:teuthology.orchestra.run.smithi051.stdout:{"epoch":4,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6824","nonce":14017},{"type":"v1","addr":"172.21.15.51:6825","nonce":14017}]},"active_addr":"172.21.15.51:6825/14017","active_change":"2020-11-09T18:08:10.549288+0000","active_mgr_features":4540138297136906239,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.6.2","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.51:0","nonce":325913836}]},{"addrvec":[{"type":"v2","addr":"172.21.15.51:0","nonce":98268959}]},{"addrvec":[{"type":"v2","addr":"172.21.15.51:0","nonce":715790231}]}]}} 2020-11-09T18:08:17.807 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2020-11-09T18:08:17.807 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2020-11-09T18:08:17.808 INFO:teuthology.orchestra.run.smithi051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2020-11-09T18:08:18.103 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:08:18.103 INFO:teuthology.orchestra.run.smithi051.stdout:{"epoch":13,"fsid":"f8cfda89-96d9-4149-8cfc-db3ea75def88","created":"2020-11-09T18:08:06.436693+0000","modified":"2020-11-09T18:08:17.511567+0000","last_up_change":"2020-11-09T18:08:11.469550+0000","last_in_change":"2020-11-09T18:08:11.469550+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-09T18:08:10.602093+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-09T18:08:14.965591+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"69041991-91cc-444e-a8c0-1444e1d4b5bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6816","nonce":14202},{"type":"v1","addr":"172.21.15.51:6817","nonce":14202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6818","nonce":14202},{"type":"v1","addr":"172.21.15.51:6819","nonce":14202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6822","nonce":14202},{"type":"v1","addr":"172.21.15.51:6823","nonce":14202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6820","nonce":14202},{"type":"v1","addr":"172.21.15.51:6821","nonce":14202}]},"public_addr":"172.21.15.51:6817/14202","cluster_addr":"172.21.15.51:6819/14202","heartbeat_back_addr":"172.21.15.51:6823/14202","heartbeat_front_addr":"172.21.15.51:6821/14202","state":["exists","up"]},{"osd":1,"uuid":"24120236-930c-4195-adc7-d15fe3d0030d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6800","nonce":14199},{"type":"v1","addr":"172.21.15.51:6801","nonce":14199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6802","nonce":14199},{"type":"v1","addr":"172.21.15.51:6804","nonce":14199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6814","nonce":14199},{"type":"v1","addr":"172.21.15.51:6815","nonce":14199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6807","nonce":14199},{"type":"v1","addr":"172.21.15.51:6810","nonce":14199}]},"public_addr":"172.21.15.51:6801/14199","cluster_addr":"172.21.15.51:6804/14199","heartbeat_back_addr":"172.21.15.51:6815/14199","heartbeat_front_addr":"172.21.15.51:6810/14199","state":["exists","up"]},{"osd":2,"uuid":"9f230dcb-16c5-4b08-a5ed-bb5ef1608640","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6803","nonce":14200},{"type":"v1","addr":"172.21.15.51:6805","nonce":14200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6806","nonce":14200},{"type":"v1","addr":"172.21.15.51:6808","nonce":14200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6812","nonce":14200},{"type":"v1","addr":"172.21.15.51:6813","nonce":14200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6809","nonce":14200},{"type":"v1","addr":"172.21.15.51:6811","nonce":14200}]},"public_addr":"172.21.15.51:6805/14200","cluster_addr":"172.21.15.51:6808/14200","heartbeat_back_addr":"172.21.15.51:6813/14200","heartbeat_front_addr":"172.21.15.51:6811/14200","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T18:08:09.769540+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T18:08:09.758329+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T18:08:09.523430+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-09T18:08:18.117 INFO:tasks.ceph.ceph_manager.c1:all up! 2020-11-09T18:08:18.118 INFO:teuthology.orchestra.run.smithi051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2020-11-09T18:08:18.413 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:08:18.413 INFO:teuthology.orchestra.run.smithi051.stdout:{"epoch":13,"fsid":"f8cfda89-96d9-4149-8cfc-db3ea75def88","created":"2020-11-09T18:08:06.436693+0000","modified":"2020-11-09T18:08:17.511567+0000","last_up_change":"2020-11-09T18:08:11.469550+0000","last_in_change":"2020-11-09T18:08:11.469550+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-09T18:08:10.602093+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-09T18:08:14.965591+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"69041991-91cc-444e-a8c0-1444e1d4b5bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6816","nonce":14202},{"type":"v1","addr":"172.21.15.51:6817","nonce":14202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6818","nonce":14202},{"type":"v1","addr":"172.21.15.51:6819","nonce":14202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6822","nonce":14202},{"type":"v1","addr":"172.21.15.51:6823","nonce":14202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6820","nonce":14202},{"type":"v1","addr":"172.21.15.51:6821","nonce":14202}]},"public_addr":"172.21.15.51:6817/14202","cluster_addr":"172.21.15.51:6819/14202","heartbeat_back_addr":"172.21.15.51:6823/14202","heartbeat_front_addr":"172.21.15.51:6821/14202","state":["exists","up"]},{"osd":1,"uuid":"24120236-930c-4195-adc7-d15fe3d0030d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6800","nonce":14199},{"type":"v1","addr":"172.21.15.51:6801","nonce":14199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6802","nonce":14199},{"type":"v1","addr":"172.21.15.51:6804","nonce":14199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6814","nonce":14199},{"type":"v1","addr":"172.21.15.51:6815","nonce":14199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6807","nonce":14199},{"type":"v1","addr":"172.21.15.51:6810","nonce":14199}]},"public_addr":"172.21.15.51:6801/14199","cluster_addr":"172.21.15.51:6804/14199","heartbeat_back_addr":"172.21.15.51:6815/14199","heartbeat_front_addr":"172.21.15.51:6810/14199","state":["exists","up"]},{"osd":2,"uuid":"9f230dcb-16c5-4b08-a5ed-bb5ef1608640","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6803","nonce":14200},{"type":"v1","addr":"172.21.15.51:6805","nonce":14200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6806","nonce":14200},{"type":"v1","addr":"172.21.15.51:6808","nonce":14200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6812","nonce":14200},{"type":"v1","addr":"172.21.15.51:6813","nonce":14200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.51:6809","nonce":14200},{"type":"v1","addr":"172.21.15.51:6811","nonce":14200}]},"public_addr":"172.21.15.51:6805/14200","cluster_addr":"172.21.15.51:6808/14200","heartbeat_back_addr":"172.21.15.51:6813/14200","heartbeat_front_addr":"172.21.15.51:6811/14200","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T18:08:09.769540+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T18:08:09.758329+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T18:08:09.523430+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-09T18:08:18.427 INFO:teuthology.orchestra.run.smithi051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2020-11-09T18:08:18.596 INFO:teuthology.orchestra.run.smithi051.stdout:34359738371 2020-11-09T18:08:18.597 INFO:teuthology.orchestra.run.smithi051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2020-11-09T18:08:18.769 INFO:teuthology.orchestra.run.smithi051.stdout:34359738371 2020-11-09T18:08:18.770 INFO:teuthology.orchestra.run.smithi051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2020-11-09T18:08:18.935 INFO:teuthology.orchestra.run.smithi051.stdout:34359738371 2020-11-09T18:08:18.935 INFO:teuthology.orchestra.run.smithi051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2020-11-09T18:08:19.228 INFO:teuthology.orchestra.run.smithi051.stdout:34359738369 2020-11-09T18:08:19.243 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738369 for osd.0 2020-11-09T18:08:20.243 INFO:teuthology.orchestra.run.smithi051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2020-11-09T18:08:20.547 INFO:teuthology.orchestra.run.smithi051.stdout:34359738370 2020-11-09T18:08:20.561 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738370 for osd.0 2020-11-09T18:08:21.562 INFO:teuthology.orchestra.run.smithi051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2020-11-09T18:08:21.865 INFO:teuthology.orchestra.run.smithi051.stdout:34359738371 2020-11-09T18:08:21.885 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738371 for osd.0 2020-11-09T18:08:21.885 INFO:teuthology.orchestra.run.smithi051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2020-11-09T18:08:22.179 INFO:teuthology.orchestra.run.smithi051.stdout:34359738371 2020-11-09T18:08:22.194 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738371 for osd.1 2020-11-09T18:08:22.194 INFO:teuthology.orchestra.run.smithi051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2020-11-09T18:08:22.483 INFO:teuthology.orchestra.run.smithi051.stdout:34359738371 2020-11-09T18:08:22.497 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738371 for osd.2 2020-11-09T18:08:22.498 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2020-11-09T18:08:22.498 INFO:teuthology.orchestra.run.smithi051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2020-11-09T18:08:22.779 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:08:22.781 INFO:teuthology.orchestra.run.smithi051.stderr:dumped all 2020-11-09T18:08:22.794 INFO:teuthology.orchestra.run.smithi051.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2020-11-09T18:08:22.558873+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14696,"kb_used_data":336,"kb_used_omap":0,"kb_used_meta":14336,"kb_avail":314558104,"statfs":{"total":322122547200,"available":322107498496,"internally_reserved":0,"allocated":344064,"data_stored":98208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14680064},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.041483"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T18:08:16.527662+0000","last_change":"2020-11-09T18:08:16.522123+0000","last_active":"2020-11-09T18:08:16.527662+0000","last_peered":"2020-11-09T18:08:16.527662+0000","last_clean":"2020-11-09T18:08:16.527662+0000","last_became_active":"2020-11-09T18:08:16.521693+0000","last_became_peered":"2020-11-09T18:08:16.521693+0000","last_unstale":"2020-11-09T18:08:16.527662+0000","last_undegraded":"2020-11-09T18:08:16.527662+0000","last_fullsized":"2020-11-09T18:08:16.527662+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T18:08:15.497487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T18:08:15.497487+0000","last_clean_scrub_stamp":"2020-11-09T18:08:15.497487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T18:08:16.527964+0000","last_change":"2020-11-09T18:08:16.522994+0000","last_active":"2020-11-09T18:08:16.527964+0000","last_peered":"2020-11-09T18:08:16.527964+0000","last_clean":"2020-11-09T18:08:16.527964+0000","last_became_active":"2020-11-09T18:08:16.522697+0000","last_became_peered":"2020-11-09T18:08:16.522697+0000","last_unstale":"2020-11-09T18:08:16.527964+0000","last_undegraded":"2020-11-09T18:08:16.527964+0000","last_fullsized":"2020-11-09T18:08:16.527964+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T18:08:15.497487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T18:08:15.497487+0000","last_clean_scrub_stamp":"2020-11-09T18:08:15.497487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T18:08:16.527634+0000","last_change":"2020-11-09T18:08:16.521865+0000","last_active":"2020-11-09T18:08:16.527634+0000","last_peered":"2020-11-09T18:08:16.527634+0000","last_clean":"2020-11-09T18:08:16.527634+0000","last_became_active":"2020-11-09T18:08:16.521441+0000","last_became_peered":"2020-11-09T18:08:16.521441+0000","last_unstale":"2020-11-09T18:08:16.527634+0000","last_undegraded":"2020-11-09T18:08:16.527634+0000","last_fullsized":"2020-11-09T18:08:16.527634+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T18:08:15.497487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T18:08:15.497487+0000","last_clean_scrub_stamp":"2020-11-09T18:08:15.497487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T18:08:16.521887+0000","last_change":"2020-11-09T18:08:16.520252+0000","last_active":"2020-11-09T18:08:16.521887+0000","last_peered":"2020-11-09T18:08:16.521887+0000","last_clean":"2020-11-09T18:08:16.521887+0000","last_became_active":"2020-11-09T18:08:16.519655+0000","last_became_peered":"2020-11-09T18:08:16.519655+0000","last_unstale":"2020-11-09T18:08:16.521887+0000","last_undegraded":"2020-11-09T18:08:16.521887+0000","last_fullsized":"2020-11-09T18:08:16.521887+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T18:08:15.497487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T18:08:15.497487+0000","last_clean_scrub_stamp":"2020-11-09T18:08:15.497487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"12","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T18:08:16.516204+0000","last_change":"2020-11-09T18:08:12.785094+0000","last_active":"2020-11-09T18:08:16.516204+0000","last_peered":"2020-11-09T18:08:16.516204+0000","last_clean":"2020-11-09T18:08:16.516204+0000","last_became_active":"2020-11-09T18:08:12.783566+0000","last_became_peered":"2020-11-09T18:08:12.783566+0000","last_unstale":"2020-11-09T18:08:16.516204+0000","last_undegraded":"2020-11-09T18:08:16.516204+0000","last_fullsized":"2020-11-09T18:08:16.516204+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":8,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T18:08:11.469550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T18:08:11.469550+0000","last_clean_scrub_stamp":"2020-11-09T18:08:11.469550+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T18:08:16.519549+0000","last_change":"2020-11-09T18:08:16.517729+0000","last_active":"2020-11-09T18:08:16.519549+0000","last_peered":"2020-11-09T18:08:16.519549+0000","last_clean":"2020-11-09T18:08:16.519549+0000","last_became_active":"2020-11-09T18:08:16.517304+0000","last_became_peered":"2020-11-09T18:08:16.517304+0000","last_unstale":"2020-11-09T18:08:16.519549+0000","last_undegraded":"2020-11-09T18:08:16.519549+0000","last_fullsized":"2020-11-09T18:08:16.519549+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T18:08:15.497487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T18:08:15.497487+0000","last_clean_scrub_stamp":"2020-11-09T18:08:15.497487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T18:08:16.522042+0000","last_change":"2020-11-09T18:08:16.520412+0000","last_active":"2020-11-09T18:08:16.522042+0000","last_peered":"2020-11-09T18:08:16.522042+0000","last_clean":"2020-11-09T18:08:16.522042+0000","last_became_active":"2020-11-09T18:08:16.519960+0000","last_became_peered":"2020-11-09T18:08:16.519960+0000","last_unstale":"2020-11-09T18:08:16.522042+0000","last_undegraded":"2020-11-09T18:08:16.522042+0000","last_fullsized":"2020-11-09T18:08:16.522042+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T18:08:15.497487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T18:08:15.497487+0000","last_clean_scrub_stamp":"2020-11-09T18:08:15.497487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T18:08:16.523146+0000","last_change":"2020-11-09T18:08:16.521227+0000","last_active":"2020-11-09T18:08:16.523146+0000","last_peered":"2020-11-09T18:08:16.523146+0000","last_clean":"2020-11-09T18:08:16.523146+0000","last_became_active":"2020-11-09T18:08:16.520878+0000","last_became_peered":"2020-11-09T18:08:16.520878+0000","last_unstale":"2020-11-09T18:08:16.523146+0000","last_undegraded":"2020-11-09T18:08:16.523146+0000","last_fullsized":"2020-11-09T18:08:16.523146+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T18:08:15.497487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T18:08:15.497487+0000","last_clean_scrub_stamp":"2020-11-09T18:08:15.497487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T18:08:16.527962+0000","last_change":"2020-11-09T18:08:16.522888+0000","last_active":"2020-11-09T18:08:16.527962+0000","last_peered":"2020-11-09T18:08:16.527962+0000","last_clean":"2020-11-09T18:08:16.527962+0000","last_became_active":"2020-11-09T18:08:16.522436+0000","last_became_peered":"2020-11-09T18:08:16.522436+0000","last_unstale":"2020-11-09T18:08:16.527962+0000","last_undegraded":"2020-11-09T18:08:16.527962+0000","last_fullsized":"2020-11-09T18:08:16.527962+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T18:08:15.497487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T18:08:15.497487+0000","last_clean_scrub_stamp":"2020-11-09T18:08:15.497487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4856,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":104852744,"statfs":{"total":107374182400,"available":107369209856,"internally_reserved":0,"allocated":114688,"data_stored":32736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94799999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.036}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":32736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.036}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.482}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":32736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76900000000000002}]}]}]}} 2020-11-09T18:08:22.796 INFO:teuthology.orchestra.run.smithi051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2020-11-09T18:08:23.085 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:08:23.088 INFO:teuthology.orchestra.run.smithi051.stderr:dumped all 2020-11-09T18:08:23.100 INFO:teuthology.orchestra.run.smithi051.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2020-11-09T18:08:22.558873+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14696,"kb_used_data":336,"kb_used_omap":0,"kb_used_meta":14336,"kb_avail":314558104,"statfs":{"total":322122547200,"available":322107498496,"internally_reserved":0,"allocated":344064,"data_stored":98208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14680064},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.041483"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T18:08:16.527662+0000","last_change":"2020-11-09T18:08:16.522123+0000","last_active":"2020-11-09T18:08:16.527662+0000","last_peered":"2020-11-09T18:08:16.527662+0000","last_clean":"2020-11-09T18:08:16.527662+0000","last_became_active":"2020-11-09T18:08:16.521693+0000","last_became_peered":"2020-11-09T18:08:16.521693+0000","last_unstale":"2020-11-09T18:08:16.527662+0000","last_undegraded":"2020-11-09T18:08:16.527662+0000","last_fullsized":"2020-11-09T18:08:16.527662+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T18:08:15.497487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T18:08:15.497487+0000","last_clean_scrub_stamp":"2020-11-09T18:08:15.497487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T18:08:16.527964+0000","last_change":"2020-11-09T18:08:16.522994+0000","last_active":"2020-11-09T18:08:16.527964+0000","last_peered":"2020-11-09T18:08:16.527964+0000","last_clean":"2020-11-09T18:08:16.527964+0000","last_became_active":"2020-11-09T18:08:16.522697+0000","last_became_peered":"2020-11-09T18:08:16.522697+0000","last_unstale":"2020-11-09T18:08:16.527964+0000","last_undegraded":"2020-11-09T18:08:16.527964+0000","last_fullsized":"2020-11-09T18:08:16.527964+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T18:08:15.497487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T18:08:15.497487+0000","last_clean_scrub_stamp":"2020-11-09T18:08:15.497487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T18:08:16.527634+0000","last_change":"2020-11-09T18:08:16.521865+0000","last_active":"2020-11-09T18:08:16.527634+0000","last_peered":"2020-11-09T18:08:16.527634+0000","last_clean":"2020-11-09T18:08:16.527634+0000","last_became_active":"2020-11-09T18:08:16.521441+0000","last_became_peered":"2020-11-09T18:08:16.521441+0000","last_unstale":"2020-11-09T18:08:16.527634+0000","last_undegraded":"2020-11-09T18:08:16.527634+0000","last_fullsized":"2020-11-09T18:08:16.527634+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T18:08:15.497487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T18:08:15.497487+0000","last_clean_scrub_stamp":"2020-11-09T18:08:15.497487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T18:08:16.521887+0000","last_change":"2020-11-09T18:08:16.520252+0000","last_active":"2020-11-09T18:08:16.521887+0000","last_peered":"2020-11-09T18:08:16.521887+0000","last_clean":"2020-11-09T18:08:16.521887+0000","last_became_active":"2020-11-09T18:08:16.519655+0000","last_became_peered":"2020-11-09T18:08:16.519655+0000","last_unstale":"2020-11-09T18:08:16.521887+0000","last_undegraded":"2020-11-09T18:08:16.521887+0000","last_fullsized":"2020-11-09T18:08:16.521887+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T18:08:15.497487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T18:08:15.497487+0000","last_clean_scrub_stamp":"2020-11-09T18:08:15.497487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"12","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T18:08:16.516204+0000","last_change":"2020-11-09T18:08:12.785094+0000","last_active":"2020-11-09T18:08:16.516204+0000","last_peered":"2020-11-09T18:08:16.516204+0000","last_clean":"2020-11-09T18:08:16.516204+0000","last_became_active":"2020-11-09T18:08:12.783566+0000","last_became_peered":"2020-11-09T18:08:12.783566+0000","last_unstale":"2020-11-09T18:08:16.516204+0000","last_undegraded":"2020-11-09T18:08:16.516204+0000","last_fullsized":"2020-11-09T18:08:16.516204+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":8,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T18:08:11.469550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T18:08:11.469550+0000","last_clean_scrub_stamp":"2020-11-09T18:08:11.469550+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T18:08:16.519549+0000","last_change":"2020-11-09T18:08:16.517729+0000","last_active":"2020-11-09T18:08:16.519549+0000","last_peered":"2020-11-09T18:08:16.519549+0000","last_clean":"2020-11-09T18:08:16.519549+0000","last_became_active":"2020-11-09T18:08:16.517304+0000","last_became_peered":"2020-11-09T18:08:16.517304+0000","last_unstale":"2020-11-09T18:08:16.519549+0000","last_undegraded":"2020-11-09T18:08:16.519549+0000","last_fullsized":"2020-11-09T18:08:16.519549+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T18:08:15.497487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T18:08:15.497487+0000","last_clean_scrub_stamp":"2020-11-09T18:08:15.497487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T18:08:16.522042+0000","last_change":"2020-11-09T18:08:16.520412+0000","last_active":"2020-11-09T18:08:16.522042+0000","last_peered":"2020-11-09T18:08:16.522042+0000","last_clean":"2020-11-09T18:08:16.522042+0000","last_became_active":"2020-11-09T18:08:16.519960+0000","last_became_peered":"2020-11-09T18:08:16.519960+0000","last_unstale":"2020-11-09T18:08:16.522042+0000","last_undegraded":"2020-11-09T18:08:16.522042+0000","last_fullsized":"2020-11-09T18:08:16.522042+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T18:08:15.497487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T18:08:15.497487+0000","last_clean_scrub_stamp":"2020-11-09T18:08:15.497487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T18:08:16.523146+0000","last_change":"2020-11-09T18:08:16.521227+0000","last_active":"2020-11-09T18:08:16.523146+0000","last_peered":"2020-11-09T18:08:16.523146+0000","last_clean":"2020-11-09T18:08:16.523146+0000","last_became_active":"2020-11-09T18:08:16.520878+0000","last_became_peered":"2020-11-09T18:08:16.520878+0000","last_unstale":"2020-11-09T18:08:16.523146+0000","last_undegraded":"2020-11-09T18:08:16.523146+0000","last_fullsized":"2020-11-09T18:08:16.523146+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T18:08:15.497487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T18:08:15.497487+0000","last_clean_scrub_stamp":"2020-11-09T18:08:15.497487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T18:08:16.527962+0000","last_change":"2020-11-09T18:08:16.522888+0000","last_active":"2020-11-09T18:08:16.527962+0000","last_peered":"2020-11-09T18:08:16.527962+0000","last_clean":"2020-11-09T18:08:16.527962+0000","last_became_active":"2020-11-09T18:08:16.522436+0000","last_became_peered":"2020-11-09T18:08:16.522436+0000","last_unstale":"2020-11-09T18:08:16.527962+0000","last_undegraded":"2020-11-09T18:08:16.527962+0000","last_fullsized":"2020-11-09T18:08:16.527962+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T18:08:15.497487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T18:08:15.497487+0000","last_clean_scrub_stamp":"2020-11-09T18:08:15.497487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4856,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":104852744,"statfs":{"total":107374182400,"available":107369209856,"internally_reserved":0,"allocated":114688,"data_stored":32736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94799999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.036}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":32736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.036}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.482}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":32736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76900000000000002}]}]}]}} 2020-11-09T18:08:23.101 INFO:tasks.ceph.ceph_manager.c1:clean! 2020-11-09T18:08:23.102 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2020-11-09T18:08:23.102 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2020-11-09T18:08:23.102 INFO:teuthology.orchestra.run.smithi051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2020-11-09T18:08:23.441 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:08:23.456 INFO:teuthology.orchestra.run.smithi051.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2020-11-09T18:08:23.456 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2020-11-09T18:08:23.457 INFO:teuthology.run_tasks:Running task ceph... 2020-11-09T18:08:23.473 INFO:tasks.ceph:Creating ceph cluster c2... 2020-11-09T18:08:23.474 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(PG_AVAILABILITY\\)'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2020-11-09T18:08:23.474 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/dang-2020-11-09_12:38:28-rgw-wip-dang-webdir-empty-object-distro-basic-smithi/5605822', 'branch': 'wip-dang-webdir-empty-object', 'description': 'rgw/multisite/{clusters frontend/civetweb omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '5605822', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'dang-2020-11-09_12:38:28-rgw-wip-dang-webdir-empty-object-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'ubuntu', 'overrides': {'admin_socket': {'branch': 'wip-dang-webdir-empty-object'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'a0560c8e5c6c33c4655e9d2a8b022793ae57419f', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': 'a0560c8e5c6c33c4655e9d2a8b022793ae57419f'}}, 'rgw': {'compression type': 'random', 'frontend': 'civetweb'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'workunit': {'branch': 'wip-dang-webdir-empty-object', 'sha1': 'a0560c8e5c6c33c4655e9d2a8b022793ae57419f'}}, 'owner': 'scheduled_dang@sidious.arb.redhat.com', 'priority': 100, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': 'a0560c8e5c6c33c4655e9d2a8b022793ae57419f', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-dang-webdir-empty-object', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_wip-dang-webdir-empty-object/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'a0560c8e5c6c33c4655e9d2a8b022793ae57419f', 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (2, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'a0560c8e5c6c33c4655e9d2a8b022793ae57419f', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'a0560c8e5c6c33c4655e9d2a8b022793ae57419f', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'tube': 'smithi', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.7071', 'targets': {'smithi051.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDChRn9ME/9o1R6oGob2yIvGJjZ3e8odesSjGsarWb9ZjpSVVq1CLSG0gjPIQwGH3lAQOoqsxc3hyAemED82m5rAOpsoeCiYN2zA5MTNVqv+oHY283y2tPAnLGcjMMIFpe/YoYBVUDoH2ppv4cpoKAD/WX31GH84Z7mXBopdTgcDZfRU6gH6YeE0/mj34rUy3Z/6oxwrNkhiMAJADRMiga/xTTAKFJQaAc4UT0JVWIp7yBdY6AVwj24vC7x35EXpJwmDwMfWAMaHxFoy/uJMazzWPyw0D7AQp857+92jl28ah1hwXTyhl4G0lqxhxX54tNsit+xwsOWwuqJpNPjUW0L', 'smithi078.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEXAj7zwy9ounBiobhGxgHVGG0io6UH6ZgSL8GxSgCz5Xjm6d8yZtJHCZRXwzH33gb18kSLU4Pc2qngB+l7qc0UMJLU31PvCo0Nz1/x2na6haA1A0UmPtd45BOsT5v1D5taYl1dN1y26UqxxeDj0NZzpgSuF1EdSKAt75H5cBJt7mu1UlDYcn0cPnsINXRm29Bp2e4X7hyj21y7QfQL6RTa3+/8T7de4l9lhjUqV/qZRepvYWmoB9pg6U0bEHgnvkS8UQefJXYMS++teARGcP1hY3KBciTDCF+WrxHkNUNyhmRUaM3wxvILElcgkVwK4vPp+AQHJ5OmtEltNe6t/Zh'}} 2020-11-09T18:08:23.475 INFO:teuthology.orchestra.run.smithi051:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2020-11-09T18:08:23.478 INFO:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2020-11-09T18:08:23.486 INFO:teuthology.orchestra.run.smithi051:> sudo install -d -m0777 -- /var/run/ceph 2020-11-09T18:08:23.526 INFO:teuthology.orchestra.run.smithi078:> sudo install -d -m0777 -- /var/run/ceph 2020-11-09T18:08:23.544 INFO:teuthology.orchestra.run.smithi078:> set -ex 2020-11-09T18:08:23.545 INFO:teuthology.orchestra.run.smithi078:> dd if=/scratch_devs of=/dev/stdout 2020-11-09T18:08:23.590 INFO:teuthology.orchestra.run.smithi078.stderr:+ dd if=/scratch_devs of=/dev/stdout 2020-11-09T18:08:23.591 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/vg_nvme/lv_1 2020-11-09T18:08:23.592 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/vg_nvme/lv_2 2020-11-09T18:08:23.592 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/vg_nvme/lv_3 2020-11-09T18:08:23.592 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/vg_nvme/lv_4 2020-11-09T18:08:23.592 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records in 2020-11-09T18:08:23.593 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records out 2020-11-09T18:08:23.593 INFO:teuthology.orchestra.run.smithi078.stderr:72 bytes copied, 0.000176793 s, 407 kB/s 2020-11-09T18:08:23.593 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2020-11-09T18:08:23.593 INFO:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-11-09T18:08:23.640 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2020-11-09T18:08:23.640 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-09T18:08:23.640 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 6h/6d Inode: 341 Links: 1 2020-11-09T18:08:23.641 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-09T18:08:23.641 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2020-11-09 18:05:13.232391384 +0000 2020-11-09T18:08:23.641 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2020-11-09 18:05:13.228393384 +0000 2020-11-09T18:08:23.641 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2020-11-09 18:05:13.228393384 +0000 2020-11-09T18:08:23.641 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2020-11-09T18:08:23.649 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2020-11-09T18:08:23.650 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2020-11-09T18:08:23.650 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000385304 s, 1.3 MB/s 2020-11-09T18:08:23.654 INFO:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-11-09T18:08:23.704 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2020-11-09T18:08:23.704 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-09T18:08:23.704 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 6h/6d Inode: 348 Links: 1 2020-11-09T18:08:23.705 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-09T18:08:23.705 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2020-11-09 18:05:13.476269377 +0000 2020-11-09T18:08:23.705 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2020-11-09 18:05:13.472271378 +0000 2020-11-09T18:08:23.705 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2020-11-09 18:05:13.472271378 +0000 2020-11-09T18:08:23.705 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2020-11-09T18:08:23.713 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2020-11-09T18:08:23.714 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2020-11-09T18:08:23.714 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000315549 s, 1.6 MB/s 2020-11-09T18:08:23.717 INFO:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-11-09T18:08:23.764 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2020-11-09T18:08:23.764 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-09T18:08:23.765 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 6h/6d Inode: 354 Links: 1 2020-11-09T18:08:23.765 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-09T18:08:23.765 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2020-11-09 18:05:13.784115372 +0000 2020-11-09T18:08:23.766 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2020-11-09 18:05:13.696159373 +0000 2020-11-09T18:08:23.766 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2020-11-09 18:05:13.696159373 +0000 2020-11-09T18:08:23.766 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2020-11-09T18:08:23.773 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2020-11-09T18:08:23.774 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2020-11-09T18:08:23.774 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000354885 s, 1.4 MB/s 2020-11-09T18:08:23.778 INFO:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-11-09T18:08:23.827 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2020-11-09T18:08:23.828 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-09T18:08:23.828 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 6h/6d Inode: 360 Links: 1 2020-11-09T18:08:23.828 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-09T18:08:23.828 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2020-11-09 18:05:14.023995366 +0000 2020-11-09T18:08:23.828 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2020-11-09 18:05:13.936039367 +0000 2020-11-09T18:08:23.829 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2020-11-09 18:05:13.936039367 +0000 2020-11-09T18:08:23.829 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2020-11-09T18:08:23.837 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2020-11-09T18:08:23.837 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2020-11-09T18:08:23.838 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000395568 s, 1.3 MB/s 2020-11-09T18:08:23.841 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-09T18:08:23.841 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi078.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2020-11-09T18:08:23.841 INFO:tasks.ceph:Generating config... 2020-11-09T18:08:23.843 INFO:tasks.ceph:[client] debug rgw = 20 2020-11-09T18:08:23.843 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2020-11-09T18:08:23.843 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2020-11-09T18:08:23.843 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-11-09T18:08:23.844 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2020-11-09T18:08:23.844 INFO:tasks.ceph:[client] rgw data log num shards = 4 2020-11-09T18:08:23.844 INFO:tasks.ceph:[client] rgw md log max shards = 4 2020-11-09T18:08:23.844 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2020-11-09T18:08:23.844 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2020-11-09T18:08:23.845 INFO:tasks.ceph:[client] setgroup = ceph 2020-11-09T18:08:23.845 INFO:tasks.ceph:[client] setuser = ceph 2020-11-09T18:08:23.845 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2020-11-09T18:08:23.845 INFO:tasks.ceph:[mgr] debug mgr = 20 2020-11-09T18:08:23.845 INFO:tasks.ceph:[mgr] debug ms = 1 2020-11-09T18:08:23.846 INFO:tasks.ceph:[mon] debug mon = 20 2020-11-09T18:08:23.846 INFO:tasks.ceph:[mon] debug ms = 1 2020-11-09T18:08:23.846 INFO:tasks.ceph:[mon] debug paxos = 20 2020-11-09T18:08:23.846 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2020-11-09T18:08:23.846 INFO:tasks.ceph:[osd] debug filestore = 20 2020-11-09T18:08:23.847 INFO:tasks.ceph:[osd] debug journal = 20 2020-11-09T18:08:23.847 INFO:tasks.ceph:[osd] debug ms = 20 2020-11-09T18:08:23.847 INFO:tasks.ceph:[osd] debug osd = 25 2020-11-09T18:08:23.847 INFO:tasks.ceph:[osd] osd fast shutdown = False 2020-11-09T18:08:23.847 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2020-11-09T18:08:23.848 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2020-11-09T18:08:23.848 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2020-11-09T18:08:23.848 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2020-11-09T18:08:23.849 INFO:tasks.ceph:Setting up c2.mon.a... 2020-11-09T18:08:23.851 INFO:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2020-11-09T18:08:23.916 INFO:teuthology.orchestra.run.smithi078.stdout:creating /etc/ceph/c2.keyring 2020-11-09T18:08:23.918 INFO:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2020-11-09T18:08:23.994 INFO:teuthology.orchestra.run.smithi078:> sudo chmod 0644 /etc/ceph/c2.keyring 2020-11-09T18:08:24.050 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.78')] 2020-11-09T18:08:24.051 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon host': '172.21.15.78', 'osd heartbeat grace': 40}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'mon osd crush smoke test': False}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2020-11-09T18:08:24.053 INFO:teuthology.orchestra.run.smithi078:> set -ex 2020-11-09T18:08:24.054 INFO:teuthology.orchestra.run.smithi078:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2020-11-09T18:08:24.097 INFO:teuthology.orchestra.run.smithi078.stderr:+ dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2020-11-09T18:08:24.098 INFO:teuthology.orchestra.run.smithi078.stderr:6+1 records in 2020-11-09T18:08:24.098 INFO:teuthology.orchestra.run.smithi078.stderr:6+1 records out 2020-11-09T18:08:24.099 INFO:teuthology.orchestra.run.smithi078.stderr:3100 bytes (3.1 kB, 3.0 KiB) copied, 8.993e-05 s, 34.5 MB/s 2020-11-09T18:08:24.099 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.78 --print /home/ubuntu/cephtest/c2.monmap 2020-11-09T18:08:24.171 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:08:24.172 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:08:24.172 INFO:teuthology.orchestra.run.smithi078.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2020-11-09T18:08:24.173 INFO:teuthology.orchestra.run.smithi078.stdout:monmaptool: generated fsid 774d91bc-658c-40ed-9c19-9f7e16b77485 2020-11-09T18:08:24.173 INFO:teuthology.orchestra.run.smithi078.stdout:epoch 0 2020-11-09T18:08:24.173 INFO:teuthology.orchestra.run.smithi078.stdout:fsid 774d91bc-658c-40ed-9c19-9f7e16b77485 2020-11-09T18:08:24.174 INFO:teuthology.orchestra.run.smithi078.stdout:last_changed 2020-11-09T18:08:24.174415+0000 2020-11-09T18:08:24.174 INFO:teuthology.orchestra.run.smithi078.stdout:created 2020-11-09T18:08:24.174415+0000 2020-11-09T18:08:24.174 INFO:teuthology.orchestra.run.smithi078.stdout:min_mon_release 0 (unknown) 2020-11-09T18:08:24.175 INFO:teuthology.orchestra.run.smithi078.stdout:election_strategy: 1 2020-11-09T18:08:24.175 INFO:teuthology.orchestra.run.smithi078.stdout:0: [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon.a 2020-11-09T18:08:24.175 INFO:teuthology.orchestra.run.smithi078.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2020-11-09T18:08:24.176 INFO:teuthology.orchestra.run.smithi078:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2020-11-09T18:08:24.223 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID 774d91bc-658c-40ed-9c19-9f7e16b77485... 2020-11-09T18:08:24.224 INFO:teuthology.orchestra.run.smithi051:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2020-11-09T18:08:24.226 INFO:teuthology.orchestra.run.smithi078:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2020-11-09T18:08:24.268 INFO:teuthology.orchestra.run.smithi051.stdout:[global] 2020-11-09T18:08:24.268 INFO:teuthology.orchestra.run.smithi051.stdout: chdir = "" 2020-11-09T18:08:24.269 INFO:teuthology.orchestra.run.smithi051.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-09T18:08:24.269 INFO:teuthology.orchestra.run.smithi051.stdout: auth supported = cephx 2020-11-09T18:08:24.269 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:08:24.269 INFO:teuthology.orchestra.run.smithi051.stdout: filestore xattr use omap = true 2020-11-09T18:08:24.269 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:08:24.269 INFO:teuthology.orchestra.run.smithi051.stdout: mon clock drift allowed = 1.000 2020-11-09T18:08:24.269 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:08:24.270 INFO:teuthology.orchestra.run.smithi051.stdout: osd crush chooseleaf type = 0 2020-11-09T18:08:24.270 INFO:teuthology.orchestra.run.smithi051.stdout: auth debug = true 2020-11-09T18:08:24.270 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:08:24.270 INFO:teuthology.orchestra.run.smithi051.stdout: ms die on old message = true 2020-11-09T18:08:24.270 INFO:teuthology.orchestra.run.smithi051.stdout: ms die on bug = true 2020-11-09T18:08:24.270 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:08:24.271 INFO:teuthology.orchestra.run.smithi051.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-09T18:08:24.271 INFO:teuthology.orchestra.run.smithi051.stdout: mon pg warn max object skew = 0 2020-11-09T18:08:24.271 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:08:24.271 INFO:teuthology.orchestra.run.smithi051.stdout: # disable pg_autoscaler by default for new pools 2020-11-09T18:08:24.271 INFO:teuthology.orchestra.run.smithi051.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-09T18:08:24.271 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:08:24.272 INFO:teuthology.orchestra.run.smithi051.stdout: osd pool default size = 2 2020-11-09T18:08:24.272 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:08:24.272 INFO:teuthology.orchestra.run.smithi051.stdout: mon osd allow primary affinity = true 2020-11-09T18:08:24.272 INFO:teuthology.orchestra.run.smithi051.stdout: mon osd allow pg remap = true 2020-11-09T18:08:24.272 INFO:teuthology.orchestra.run.smithi051.stdout: mon warn on legacy crush tunables = false 2020-11-09T18:08:24.272 INFO:teuthology.orchestra.run.smithi051.stdout: mon warn on crush straw calc version zero = false 2020-11-09T18:08:24.272 INFO:teuthology.orchestra.run.smithi051.stdout: mon warn on no sortbitwise = false 2020-11-09T18:08:24.273 INFO:teuthology.orchestra.run.smithi051.stdout: mon warn on osd down out interval zero = false 2020-11-09T18:08:24.273 INFO:teuthology.orchestra.run.smithi051.stdout: mon warn on too few osds = false 2020-11-09T18:08:24.273 INFO:teuthology.orchestra.run.smithi051.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-09T18:08:24.273 INFO:teuthology.orchestra.run.smithi051.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-09T18:08:24.273 INFO:teuthology.orchestra.run.smithi051.stdout: mon_allow_pool_size_one = true 2020-11-09T18:08:24.273 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:08:24.273 INFO:teuthology.orchestra.run.smithi051.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-09T18:08:24.274 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:08:24.274 INFO:teuthology.orchestra.run.smithi051.stdout: osd default data pool replay window = 5 2020-11-09T18:08:24.274 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:08:24.274 INFO:teuthology.orchestra.run.smithi051.stdout: mon allow pool delete = true 2020-11-09T18:08:24.274 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:08:24.274 INFO:teuthology.orchestra.run.smithi051.stdout: mon cluster log file level = debug 2020-11-09T18:08:24.274 INFO:teuthology.orchestra.run.smithi051.stdout: debug asserts on shutdown = true 2020-11-09T18:08:24.275 INFO:teuthology.orchestra.run.smithi051.stdout: mon host = 172.21.15.78 2020-11-09T18:08:24.275 INFO:teuthology.orchestra.run.smithi051.stdout: osd heartbeat grace = 40 2020-11-09T18:08:24.275 INFO:teuthology.orchestra.run.smithi051.stdout: fsid = 774d91bc-658c-40ed-9c19-9f7e16b77485 2020-11-09T18:08:24.275 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:08:24.275 INFO:teuthology.orchestra.run.smithi051.stdout:[osd] 2020-11-09T18:08:24.275 INFO:teuthology.orchestra.run.smithi051.stdout: osd journal size = 100 2020-11-09T18:08:24.275 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:08:24.275 INFO:teuthology.orchestra.run.smithi051.stdout: osd scrub load threshold = 5.0 2020-11-09T18:08:24.276 INFO:teuthology.orchestra.run.smithi051.stdout: osd scrub max interval = 600 2020-11-09T18:08:24.276 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:08:24.276 INFO:teuthology.orchestra.run.smithi051.stdout: osd recover clone overlap = true 2020-11-09T18:08:24.276 INFO:teuthology.orchestra.run.smithi051.stdout: osd recovery max chunk = 1048576 2020-11-09T18:08:24.276 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:08:24.276 INFO:teuthology.orchestra.run.smithi051.stdout: osd debug shutdown = true 2020-11-09T18:08:24.276 INFO:teuthology.orchestra.run.smithi051.stdout: osd debug op order = true 2020-11-09T18:08:24.276 INFO:teuthology.orchestra.run.smithi051.stdout: osd debug verify stray on activate = true 2020-11-09T18:08:24.277 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:08:24.277 INFO:teuthology.orchestra.run.smithi051.stdout: osd open classes on start = true 2020-11-09T18:08:24.277 INFO:teuthology.orchestra.run.smithi051.stdout: osd debug pg log writeout = true 2020-11-09T18:08:24.277 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:08:24.277 INFO:teuthology.orchestra.run.smithi051.stdout: osd deep scrub update digest min age = 30 2020-11-09T18:08:24.277 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:08:24.277 INFO:teuthology.orchestra.run.smithi051.stdout: osd map max advance = 10 2020-11-09T18:08:24.278 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:08:24.278 INFO:teuthology.orchestra.run.smithi051.stdout: journal zero on create = true 2020-11-09T18:08:24.278 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:08:24.278 INFO:teuthology.orchestra.run.smithi051.stdout: filestore ondisk finisher threads = 3 2020-11-09T18:08:24.278 INFO:teuthology.orchestra.run.smithi051.stdout: filestore apply finisher threads = 3 2020-11-09T18:08:24.278 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:08:24.278 INFO:teuthology.orchestra.run.smithi051.stdout: bdev debug aio = true 2020-11-09T18:08:24.279 INFO:teuthology.orchestra.run.smithi051.stdout: osd debug misdirected ops = true 2020-11-09T18:08:24.279 INFO:teuthology.orchestra.run.smithi051.stdout: debug filestore = 20 2020-11-09T18:08:24.279 INFO:teuthology.orchestra.run.smithi051.stdout: debug journal = 20 2020-11-09T18:08:24.279 INFO:teuthology.orchestra.run.smithi051.stdout: debug ms = 20 2020-11-09T18:08:24.279 INFO:teuthology.orchestra.run.smithi051.stdout: debug osd = 25 2020-11-09T18:08:24.279 INFO:teuthology.orchestra.run.smithi051.stdout: osd fast shutdown = False 2020-11-09T18:08:24.280 INFO:teuthology.orchestra.run.smithi051.stdout: rocksdb delete range threshold = 0 2020-11-09T18:08:24.280 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:08:24.280 INFO:teuthology.orchestra.run.smithi051.stdout:[mgr] 2020-11-09T18:08:24.280 INFO:teuthology.orchestra.run.smithi051.stdout: debug ms = 1 2020-11-09T18:08:24.280 INFO:teuthology.orchestra.run.smithi051.stdout: debug mgr = 20 2020-11-09T18:08:24.281 INFO:teuthology.orchestra.run.smithi051.stdout: debug mon = 20 2020-11-09T18:08:24.281 INFO:teuthology.orchestra.run.smithi051.stdout: debug auth = 20 2020-11-09T18:08:24.281 INFO:teuthology.orchestra.run.smithi051.stdout: mon reweight min pgs per osd = 4 2020-11-09T18:08:24.281 INFO:teuthology.orchestra.run.smithi051.stdout: mon reweight min bytes per osd = 10 2020-11-09T18:08:24.282 INFO:teuthology.orchestra.run.smithi051.stdout: mgr/telemetry/nag = false 2020-11-09T18:08:24.282 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:08:24.282 INFO:teuthology.orchestra.run.smithi051.stdout:[mon] 2020-11-09T18:08:24.282 INFO:teuthology.orchestra.run.smithi051.stdout: debug ms = 1 2020-11-09T18:08:24.283 INFO:teuthology.orchestra.run.smithi051.stdout: debug mon = 20 2020-11-09T18:08:24.283 INFO:teuthology.orchestra.run.smithi051.stdout: debug paxos = 20 2020-11-09T18:08:24.283 INFO:teuthology.orchestra.run.smithi051.stdout: debug auth = 20 2020-11-09T18:08:24.283 INFO:teuthology.orchestra.run.smithi051.stdout: mon data avail warn = 5 2020-11-09T18:08:24.283 INFO:teuthology.orchestra.run.smithi051.stdout: mon mgr mkfs grace = 240 2020-11-09T18:08:24.283 INFO:teuthology.orchestra.run.smithi051.stdout: mon reweight min pgs per osd = 4 2020-11-09T18:08:24.284 INFO:teuthology.orchestra.run.smithi051.stdout: mon osd reporter subtree level = osd 2020-11-09T18:08:24.284 INFO:teuthology.orchestra.run.smithi051.stdout: mon osd prime pg temp = true 2020-11-09T18:08:24.284 INFO:teuthology.orchestra.run.smithi051.stdout: mon reweight min bytes per osd = 10 2020-11-09T18:08:24.284 INFO:teuthology.orchestra.run.smithi051.stdout: mon osd crush smoke test = False 2020-11-09T18:08:24.284 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:08:24.284 INFO:teuthology.orchestra.run.smithi051.stdout:[client] 2020-11-09T18:08:24.284 INFO:teuthology.orchestra.run.smithi051.stdout: rgw cache enabled = true 2020-11-09T18:08:24.285 INFO:teuthology.orchestra.run.smithi051.stdout: rgw enable ops log = true 2020-11-09T18:08:24.285 INFO:teuthology.orchestra.run.smithi051.stdout: rgw enable usage log = true 2020-11-09T18:08:24.285 INFO:teuthology.orchestra.run.smithi051.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-09T18:08:24.285 INFO:teuthology.orchestra.run.smithi051.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-09T18:08:24.285 INFO:teuthology.orchestra.run.smithi051.stdout: debug rgw = 20 2020-11-09T18:08:24.285 INFO:teuthology.orchestra.run.smithi051.stdout: rgw crypt require ssl = False 2020-11-09T18:08:24.285 INFO:teuthology.orchestra.run.smithi051.stdout: rgw crypt s3 kms backend = testing 2020-11-09T18:08:24.285 INFO:teuthology.orchestra.run.smithi051.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-11-09T18:08:24.286 INFO:teuthology.orchestra.run.smithi051.stdout: rgw curl low speed time = 300 2020-11-09T18:08:24.286 INFO:teuthology.orchestra.run.smithi051.stdout: rgw data log num shards = 4 2020-11-09T18:08:24.286 INFO:teuthology.orchestra.run.smithi051.stdout: rgw md log max shards = 4 2020-11-09T18:08:24.286 INFO:teuthology.orchestra.run.smithi051.stdout: rgw sync log trim interval = 0 2020-11-09T18:08:24.286 INFO:teuthology.orchestra.run.smithi051.stdout: rgw sync obj etag verify = True 2020-11-09T18:08:24.286 INFO:teuthology.orchestra.run.smithi051.stdout: setgroup = ceph 2020-11-09T18:08:24.287 INFO:teuthology.orchestra.run.smithi051.stdout: setuser = ceph 2020-11-09T18:08:24.287 INFO:teuthology.orchestra.run.smithi051.stdout:[mon.a] 2020-11-09T18:08:24.287 INFO:teuthology.orchestra.run.smithi051.stdout:[osd.0] 2020-11-09T18:08:24.287 INFO:teuthology.orchestra.run.smithi051.stdout: osd_max_omap_entries_per_request = 10 2020-11-09T18:08:24.287 INFO:teuthology.orchestra.run.smithi051.stdout:[osd.1] 2020-11-09T18:08:24.287 INFO:teuthology.orchestra.run.smithi051.stdout: osd_max_omap_entries_per_request = 1000 2020-11-09T18:08:24.288 INFO:teuthology.orchestra.run.smithi051.stdout:[osd.2] 2020-11-09T18:08:24.288 INFO:teuthology.orchestra.run.smithi051.stdout: osd_max_omap_entries_per_request = 10000 2020-11-09T18:08:24.296 INFO:teuthology.orchestra.run.smithi078.stdout:[global] 2020-11-09T18:08:24.297 INFO:teuthology.orchestra.run.smithi078.stdout: chdir = "" 2020-11-09T18:08:24.297 INFO:teuthology.orchestra.run.smithi078.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-09T18:08:24.297 INFO:teuthology.orchestra.run.smithi078.stdout: auth supported = cephx 2020-11-09T18:08:24.297 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:08:24.298 INFO:teuthology.orchestra.run.smithi078.stdout: filestore xattr use omap = true 2020-11-09T18:08:24.298 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:08:24.298 INFO:teuthology.orchestra.run.smithi078.stdout: mon clock drift allowed = 1.000 2020-11-09T18:08:24.298 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:08:24.298 INFO:teuthology.orchestra.run.smithi078.stdout: osd crush chooseleaf type = 0 2020-11-09T18:08:24.298 INFO:teuthology.orchestra.run.smithi078.stdout: auth debug = true 2020-11-09T18:08:24.299 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:08:24.299 INFO:teuthology.orchestra.run.smithi078.stdout: ms die on old message = true 2020-11-09T18:08:24.299 INFO:teuthology.orchestra.run.smithi078.stdout: ms die on bug = true 2020-11-09T18:08:24.299 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:08:24.299 INFO:teuthology.orchestra.run.smithi078.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-09T18:08:24.299 INFO:teuthology.orchestra.run.smithi078.stdout: mon pg warn max object skew = 0 2020-11-09T18:08:24.299 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:08:24.299 INFO:teuthology.orchestra.run.smithi078.stdout: # disable pg_autoscaler by default for new pools 2020-11-09T18:08:24.300 INFO:teuthology.orchestra.run.smithi078.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-09T18:08:24.300 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:08:24.300 INFO:teuthology.orchestra.run.smithi078.stdout: osd pool default size = 2 2020-11-09T18:08:24.300 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:08:24.300 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd allow primary affinity = true 2020-11-09T18:08:24.300 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd allow pg remap = true 2020-11-09T18:08:24.300 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on legacy crush tunables = false 2020-11-09T18:08:24.301 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on crush straw calc version zero = false 2020-11-09T18:08:24.301 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on no sortbitwise = false 2020-11-09T18:08:24.301 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on osd down out interval zero = false 2020-11-09T18:08:24.301 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on too few osds = false 2020-11-09T18:08:24.301 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-09T18:08:24.301 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-09T18:08:24.301 INFO:teuthology.orchestra.run.smithi078.stdout: mon_allow_pool_size_one = true 2020-11-09T18:08:24.301 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:08:24.302 INFO:teuthology.orchestra.run.smithi078.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-09T18:08:24.302 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:08:24.302 INFO:teuthology.orchestra.run.smithi078.stdout: osd default data pool replay window = 5 2020-11-09T18:08:24.302 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:08:24.302 INFO:teuthology.orchestra.run.smithi078.stdout: mon allow pool delete = true 2020-11-09T18:08:24.302 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:08:24.302 INFO:teuthology.orchestra.run.smithi078.stdout: mon cluster log file level = debug 2020-11-09T18:08:24.303 INFO:teuthology.orchestra.run.smithi078.stdout: debug asserts on shutdown = true 2020-11-09T18:08:24.303 INFO:teuthology.orchestra.run.smithi078.stdout: mon host = 172.21.15.78 2020-11-09T18:08:24.303 INFO:teuthology.orchestra.run.smithi078.stdout: osd heartbeat grace = 40 2020-11-09T18:08:24.303 INFO:teuthology.orchestra.run.smithi078.stdout: fsid = 774d91bc-658c-40ed-9c19-9f7e16b77485 2020-11-09T18:08:24.303 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:08:24.303 INFO:teuthology.orchestra.run.smithi078.stdout:[osd] 2020-11-09T18:08:24.303 INFO:teuthology.orchestra.run.smithi078.stdout: osd journal size = 100 2020-11-09T18:08:24.303 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:08:24.304 INFO:teuthology.orchestra.run.smithi078.stdout: osd scrub load threshold = 5.0 2020-11-09T18:08:24.304 INFO:teuthology.orchestra.run.smithi078.stdout: osd scrub max interval = 600 2020-11-09T18:08:24.304 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:08:24.304 INFO:teuthology.orchestra.run.smithi078.stdout: osd recover clone overlap = true 2020-11-09T18:08:24.304 INFO:teuthology.orchestra.run.smithi078.stdout: osd recovery max chunk = 1048576 2020-11-09T18:08:24.304 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:08:24.304 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug shutdown = true 2020-11-09T18:08:24.305 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug op order = true 2020-11-09T18:08:24.305 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug verify stray on activate = true 2020-11-09T18:08:24.305 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:08:24.305 INFO:teuthology.orchestra.run.smithi078.stdout: osd open classes on start = true 2020-11-09T18:08:24.305 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug pg log writeout = true 2020-11-09T18:08:24.305 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:08:24.305 INFO:teuthology.orchestra.run.smithi078.stdout: osd deep scrub update digest min age = 30 2020-11-09T18:08:24.306 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:08:24.306 INFO:teuthology.orchestra.run.smithi078.stdout: osd map max advance = 10 2020-11-09T18:08:24.306 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:08:24.306 INFO:teuthology.orchestra.run.smithi078.stdout: journal zero on create = true 2020-11-09T18:08:24.306 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:08:24.306 INFO:teuthology.orchestra.run.smithi078.stdout: filestore ondisk finisher threads = 3 2020-11-09T18:08:24.307 INFO:teuthology.orchestra.run.smithi078.stdout: filestore apply finisher threads = 3 2020-11-09T18:08:24.307 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:08:24.307 INFO:teuthology.orchestra.run.smithi078.stdout: bdev debug aio = true 2020-11-09T18:08:24.307 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug misdirected ops = true 2020-11-09T18:08:24.307 INFO:teuthology.orchestra.run.smithi078.stdout: debug filestore = 20 2020-11-09T18:08:24.307 INFO:teuthology.orchestra.run.smithi078.stdout: debug journal = 20 2020-11-09T18:08:24.308 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 20 2020-11-09T18:08:24.308 INFO:teuthology.orchestra.run.smithi078.stdout: debug osd = 25 2020-11-09T18:08:24.308 INFO:teuthology.orchestra.run.smithi078.stdout: osd fast shutdown = False 2020-11-09T18:08:24.308 INFO:teuthology.orchestra.run.smithi078.stdout: rocksdb delete range threshold = 0 2020-11-09T18:08:24.308 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:08:24.309 INFO:teuthology.orchestra.run.smithi078.stdout:[mgr] 2020-11-09T18:08:24.309 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 1 2020-11-09T18:08:24.309 INFO:teuthology.orchestra.run.smithi078.stdout: debug mgr = 20 2020-11-09T18:08:24.309 INFO:teuthology.orchestra.run.smithi078.stdout: debug mon = 20 2020-11-09T18:08:24.309 INFO:teuthology.orchestra.run.smithi078.stdout: debug auth = 20 2020-11-09T18:08:24.309 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min pgs per osd = 4 2020-11-09T18:08:24.310 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min bytes per osd = 10 2020-11-09T18:08:24.310 INFO:teuthology.orchestra.run.smithi078.stdout: mgr/telemetry/nag = false 2020-11-09T18:08:24.310 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:08:24.310 INFO:teuthology.orchestra.run.smithi078.stdout:[mon] 2020-11-09T18:08:24.310 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 1 2020-11-09T18:08:24.311 INFO:teuthology.orchestra.run.smithi078.stdout: debug mon = 20 2020-11-09T18:08:24.311 INFO:teuthology.orchestra.run.smithi078.stdout: debug paxos = 20 2020-11-09T18:08:24.311 INFO:teuthology.orchestra.run.smithi078.stdout: debug auth = 20 2020-11-09T18:08:24.311 INFO:teuthology.orchestra.run.smithi078.stdout: mon data avail warn = 5 2020-11-09T18:08:24.311 INFO:teuthology.orchestra.run.smithi078.stdout: mon mgr mkfs grace = 240 2020-11-09T18:08:24.311 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min pgs per osd = 4 2020-11-09T18:08:24.311 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd reporter subtree level = osd 2020-11-09T18:08:24.311 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd prime pg temp = true 2020-11-09T18:08:24.312 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min bytes per osd = 10 2020-11-09T18:08:24.312 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd crush smoke test = False 2020-11-09T18:08:24.312 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:08:24.312 INFO:teuthology.orchestra.run.smithi078.stdout:[client] 2020-11-09T18:08:24.312 INFO:teuthology.orchestra.run.smithi078.stdout: rgw cache enabled = true 2020-11-09T18:08:24.312 INFO:teuthology.orchestra.run.smithi078.stdout: rgw enable ops log = true 2020-11-09T18:08:24.312 INFO:teuthology.orchestra.run.smithi078.stdout: rgw enable usage log = true 2020-11-09T18:08:24.313 INFO:teuthology.orchestra.run.smithi078.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-09T18:08:24.313 INFO:teuthology.orchestra.run.smithi078.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-09T18:08:24.313 INFO:teuthology.orchestra.run.smithi078.stdout: debug rgw = 20 2020-11-09T18:08:24.313 INFO:teuthology.orchestra.run.smithi078.stdout: rgw crypt require ssl = False 2020-11-09T18:08:24.313 INFO:teuthology.orchestra.run.smithi078.stdout: rgw crypt s3 kms backend = testing 2020-11-09T18:08:24.313 INFO:teuthology.orchestra.run.smithi078.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-11-09T18:08:24.313 INFO:teuthology.orchestra.run.smithi078.stdout: rgw curl low speed time = 300 2020-11-09T18:08:24.314 INFO:teuthology.orchestra.run.smithi078.stdout: rgw data log num shards = 4 2020-11-09T18:08:24.314 INFO:teuthology.orchestra.run.smithi078.stdout: rgw md log max shards = 4 2020-11-09T18:08:24.314 INFO:teuthology.orchestra.run.smithi078.stdout: rgw sync log trim interval = 0 2020-11-09T18:08:24.314 INFO:teuthology.orchestra.run.smithi078.stdout: rgw sync obj etag verify = True 2020-11-09T18:08:24.314 INFO:teuthology.orchestra.run.smithi078.stdout: setgroup = ceph 2020-11-09T18:08:24.314 INFO:teuthology.orchestra.run.smithi078.stdout: setuser = ceph 2020-11-09T18:08:24.314 INFO:teuthology.orchestra.run.smithi078.stdout:[mon.a] 2020-11-09T18:08:24.314 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.0] 2020-11-09T18:08:24.315 INFO:teuthology.orchestra.run.smithi078.stdout: osd_max_omap_entries_per_request = 10 2020-11-09T18:08:24.315 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.1] 2020-11-09T18:08:24.315 INFO:teuthology.orchestra.run.smithi078.stdout: osd_max_omap_entries_per_request = 1000 2020-11-09T18:08:24.315 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.2] 2020-11-09T18:08:24.315 INFO:teuthology.orchestra.run.smithi078.stdout: osd_max_omap_entries_per_request = 10000 2020-11-09T18:08:24.316 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2020-11-09T18:08:24.317 INFO:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2020-11-09T18:08:24.405 INFO:tasks.ceph:Copying monmap to all nodes... 2020-11-09T18:08:24.406 INFO:teuthology.orchestra.run.smithi078:> set -ex 2020-11-09T18:08:24.406 INFO:teuthology.orchestra.run.smithi078:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2020-11-09T18:08:24.410 INFO:teuthology.orchestra.run.smithi078.stderr:+ dd if=/etc/ceph/c2.keyring of=/dev/stdout 2020-11-09T18:08:24.411 INFO:teuthology.orchestra.run.smithi078.stdout:[mon.] 2020-11-09T18:08:24.411 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQCXhalfKcFHOxAAQoz3JITv+mD7Cm1XiFl6kQ== 2020-11-09T18:08:24.411 INFO:teuthology.orchestra.run.smithi078.stdout:[client.admin] 2020-11-09T18:08:24.411 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQCYhalfIn0vGBAA7+LJnhZ5vRgG7RO/KRcJcQ== 2020-11-09T18:08:24.412 INFO:teuthology.orchestra.run.smithi078.stdout: caps mds = "allow *" 2020-11-09T18:08:24.412 INFO:teuthology.orchestra.run.smithi078.stdout: caps mgr = "allow *" 2020-11-09T18:08:24.412 INFO:teuthology.orchestra.run.smithi078.stdout: caps mon = "allow *" 2020-11-09T18:08:24.412 INFO:teuthology.orchestra.run.smithi078.stdout: caps osd = "allow *" 2020-11-09T18:08:24.412 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records in 2020-11-09T18:08:24.412 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records out 2020-11-09T18:08:24.414 INFO:teuthology.orchestra.run.smithi078.stderr:206 bytes copied, 7.9463e-05 s, 2.6 MB/s 2020-11-09T18:08:24.414 INFO:teuthology.orchestra.run.smithi078:> set -ex 2020-11-09T18:08:24.415 INFO:teuthology.orchestra.run.smithi078:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2020-11-09T18:08:24.457 INFO:teuthology.orchestra.run.smithi078.stderr:+ dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2020-11-09T18:08:24.459 INFO:teuthology.orchestra.run.smithi078.stdout: �wM��e�@��~�t����_``e 2020-11-09T18:08:24.459 INFO:teuthology.orchestra.run.smithi078.stdout:���_``e 2020-11-09T18:08:24.459 INFO:teuthology.orchestra.run.smithi078.stdout:�aXa �N��Na 2020-11-09T18:08:24.459 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records in 2020-11-09T18:08:24.460 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records out 2020-11-09T18:08:24.460 INFO:teuthology.orchestra.run.smithi078.stderr:201 bytes copied, 0.000146966 s, 1.4 MB/s 2020-11-09T18:08:24.460 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi051.front.sepia.ceph.com 2020-11-09T18:08:24.460 INFO:teuthology.orchestra.run.smithi051:> set -ex 2020-11-09T18:08:24.460 INFO:teuthology.orchestra.run.smithi051:> sudo dd of=/etc/ceph/c2.keyring 2020-11-09T18:08:24.461 INFO:teuthology.orchestra.run.smithi051:> sudo chmod 0644 /etc/ceph/c2.keyring 2020-11-09T18:08:24.466 INFO:teuthology.orchestra.run.smithi051.stderr:+ sudo dd of=/etc/ceph/c2.keyring 2020-11-09T18:08:24.475 INFO:teuthology.orchestra.run.smithi051.stderr:0+1 records in 2020-11-09T18:08:24.476 INFO:teuthology.orchestra.run.smithi051.stderr:0+1 records out 2020-11-09T18:08:24.476 INFO:teuthology.orchestra.run.smithi051.stderr:206 bytes copied, 7.603e-05 s, 2.7 MB/s 2020-11-09T18:08:24.476 INFO:teuthology.orchestra.run.smithi051.stderr:+ sudo chmod 0644 /etc/ceph/c2.keyring 2020-11-09T18:08:24.486 INFO:teuthology.orchestra.run.smithi051:> set -ex 2020-11-09T18:08:24.486 INFO:teuthology.orchestra.run.smithi051:> dd of=/home/ubuntu/cephtest/c2.monmap 2020-11-09T18:08:24.533 INFO:teuthology.orchestra.run.smithi051.stderr:+ dd of=/home/ubuntu/cephtest/c2.monmap 2020-11-09T18:08:24.534 INFO:teuthology.orchestra.run.smithi051.stderr:0+1 records in 2020-11-09T18:08:24.535 INFO:teuthology.orchestra.run.smithi051.stderr:0+1 records out 2020-11-09T18:08:24.535 INFO:teuthology.orchestra.run.smithi051.stderr:201 bytes copied, 8.169e-05 s, 2.5 MB/s 2020-11-09T18:08:24.535 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi078.front.sepia.ceph.com 2020-11-09T18:08:24.535 INFO:teuthology.orchestra.run.smithi078:> set -ex 2020-11-09T18:08:24.536 INFO:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/ceph/c2.keyring 2020-11-09T18:08:24.536 INFO:teuthology.orchestra.run.smithi078:> sudo chmod 0644 /etc/ceph/c2.keyring 2020-11-09T18:08:24.541 INFO:teuthology.orchestra.run.smithi078.stderr:+ sudo dd of=/etc/ceph/c2.keyring 2020-11-09T18:08:24.550 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records in 2020-11-09T18:08:24.550 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records out 2020-11-09T18:08:24.550 INFO:teuthology.orchestra.run.smithi078.stderr:206 bytes copied, 0.00011953 s, 1.7 MB/s 2020-11-09T18:08:24.552 INFO:teuthology.orchestra.run.smithi078.stderr:+ sudo chmod 0644 /etc/ceph/c2.keyring 2020-11-09T18:08:24.560 INFO:teuthology.orchestra.run.smithi078:> set -ex 2020-11-09T18:08:24.561 INFO:teuthology.orchestra.run.smithi078:> dd of=/home/ubuntu/cephtest/c2.monmap 2020-11-09T18:08:24.605 INFO:teuthology.orchestra.run.smithi078.stderr:+ dd of=/home/ubuntu/cephtest/c2.monmap 2020-11-09T18:08:24.607 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records in 2020-11-09T18:08:24.607 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records out 2020-11-09T18:08:24.607 INFO:teuthology.orchestra.run.smithi078.stderr:201 bytes copied, 0.000127583 s, 1.6 MB/s 2020-11-09T18:08:24.607 INFO:tasks.ceph:Setting up mon nodes... 2020-11-09T18:08:24.608 INFO:tasks.ceph:Setting up mgr nodes... 2020-11-09T18:08:24.608 INFO:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2020-11-09T18:08:24.690 INFO:teuthology.orchestra.run.smithi078.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2020-11-09T18:08:24.692 INFO:tasks.ceph:Setting up mds nodes... 2020-11-09T18:08:24.692 INFO:tasks.ceph_client:Setting up client nodes... 2020-11-09T18:08:24.693 INFO:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2020-11-09T18:08:24.765 INFO:teuthology.orchestra.run.smithi078.stdout:creating /etc/ceph/c2.client.0.keyring 2020-11-09T18:08:24.777 INFO:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2020-11-09T18:08:24.849 INFO:teuthology.orchestra.run.smithi078.stdout:creating /etc/ceph/c2.client.1.keyring 2020-11-09T18:08:24.861 INFO:tasks.ceph:Running mkfs on osd nodes... 2020-11-09T18:08:24.861 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi051.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi078.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2020-11-09T18:08:24.861 INFO:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2020-11-09T18:08:24.915 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-09T18:08:24.916 INFO:tasks.ceph:role: c2.osd.0 2020-11-09T18:08:24.916 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi078.front.sepia.ceph.com 2020-11-09T18:08:24.916 INFO:teuthology.orchestra.run.smithi078:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-11-09T18:08:25.436 INFO:teuthology.orchestra.run.smithi078.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-11-09T18:08:25.436 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-09T18:08:25.436 INFO:teuthology.orchestra.run.smithi078.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-11-09T18:08:25.436 INFO:teuthology.orchestra.run.smithi078.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-09T18:08:25.436 INFO:teuthology.orchestra.run.smithi078.stdout: = sunit=0 swidth=0 blks 2020-11-09T18:08:25.437 INFO:teuthology.orchestra.run.smithi078.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-11-09T18:08:25.437 INFO:teuthology.orchestra.run.smithi078.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-09T18:08:25.437 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-09T18:08:25.437 INFO:teuthology.orchestra.run.smithi078.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-09T18:08:25.438 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi078.front.sepia.ceph.com -o noatime 2020-11-09T18:08:25.438 INFO:teuthology.orchestra.run.smithi078:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2020-11-09T18:08:25.514 INFO:teuthology.orchestra.run.smithi078:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2020-11-09T18:08:25.525 INFO:teuthology.orchestra.run.smithi078.stderr:sudo: /sbin/restorecon: command not found 2020-11-09T18:08:25.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-09T18:08:25.526 INFO:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2020-11-09T18:08:25.584 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-09T18:08:25.584 INFO:tasks.ceph:role: c2.osd.1 2020-11-09T18:08:25.584 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi078.front.sepia.ceph.com 2020-11-09T18:08:25.585 INFO:teuthology.orchestra.run.smithi078:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-11-09T18:08:26.098 INFO:teuthology.orchestra.run.smithi078.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-11-09T18:08:26.098 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-09T18:08:26.098 INFO:teuthology.orchestra.run.smithi078.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-11-09T18:08:26.098 INFO:teuthology.orchestra.run.smithi078.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-09T18:08:26.099 INFO:teuthology.orchestra.run.smithi078.stdout: = sunit=0 swidth=0 blks 2020-11-09T18:08:26.099 INFO:teuthology.orchestra.run.smithi078.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-11-09T18:08:26.099 INFO:teuthology.orchestra.run.smithi078.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-09T18:08:26.099 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-09T18:08:26.099 INFO:teuthology.orchestra.run.smithi078.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-09T18:08:26.100 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi078.front.sepia.ceph.com -o noatime 2020-11-09T18:08:26.100 INFO:teuthology.orchestra.run.smithi078:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2020-11-09T18:08:26.173 INFO:teuthology.orchestra.run.smithi078:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2020-11-09T18:08:26.183 INFO:teuthology.orchestra.run.smithi078.stderr:sudo: /sbin/restorecon: command not found 2020-11-09T18:08:26.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-09T18:08:26.565 INFO:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2020-11-09T18:08:26.620 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-09T18:08:26.620 INFO:tasks.ceph:role: c2.osd.2 2020-11-09T18:08:26.620 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi078.front.sepia.ceph.com 2020-11-09T18:08:26.621 INFO:teuthology.orchestra.run.smithi078:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2020-11-09T18:08:27.138 INFO:teuthology.orchestra.run.smithi078.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2020-11-09T18:08:27.139 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-09T18:08:27.139 INFO:teuthology.orchestra.run.smithi078.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-11-09T18:08:27.139 INFO:teuthology.orchestra.run.smithi078.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-09T18:08:27.139 INFO:teuthology.orchestra.run.smithi078.stdout: = sunit=0 swidth=0 blks 2020-11-09T18:08:27.139 INFO:teuthology.orchestra.run.smithi078.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-11-09T18:08:27.140 INFO:teuthology.orchestra.run.smithi078.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-09T18:08:27.140 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-09T18:08:27.140 INFO:teuthology.orchestra.run.smithi078.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-09T18:08:27.141 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi078.front.sepia.ceph.com -o noatime 2020-11-09T18:08:27.141 INFO:teuthology.orchestra.run.smithi078:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2020-11-09T18:08:27.184 INFO:teuthology.orchestra.run.smithi078:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2020-11-09T18:08:27.236 INFO:teuthology.orchestra.run.smithi078.stderr:sudo: /sbin/restorecon: command not found 2020-11-09T18:08:27.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-09T18:08:27.237 INFO:teuthology.orchestra.run.smithi078:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2020-11-09T18:08:27.373 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:08:27.371+0000 7fa7546dbd80 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2020-11-09T18:08:27.375 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:08:27.375+0000 7fa7546dbd80 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2020-11-09T18:08:27.376 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:08:27.375+0000 7fa7546dbd80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2020-11-09T18:08:27.376 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:08:27.375+0000 7fa7546dbd80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2020-11-09T18:08:27.376 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:08:27.375+0000 7fa7546dbd80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2020-11-09T18:08:27.379 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:08:27.379+0000 7fa7546dbd80 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2020-11-09T18:08:29.561 INFO:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2020-11-09T18:08:29.577 INFO:teuthology.orchestra.run.smithi078:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2020-11-09T18:08:29.660 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:08:29.659+0000 7f070db2fd80 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2020-11-09T18:08:29.660 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:08:29.659+0000 7f070db2fd80 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2020-11-09T18:08:29.660 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:08:29.659+0000 7f070db2fd80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2020-11-09T18:08:29.660 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:08:29.659+0000 7f070db2fd80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2020-11-09T18:08:29.660 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:08:29.659+0000 7f070db2fd80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2020-11-09T18:08:29.661 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:08:29.659+0000 7f070db2fd80 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2020-11-09T18:08:31.837 INFO:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2020-11-09T18:08:31.896 INFO:teuthology.orchestra.run.smithi078:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2020-11-09T18:08:31.976 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:08:31.975+0000 7efca86b8d80 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2020-11-09T18:08:31.976 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:08:31.975+0000 7efca86b8d80 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2020-11-09T18:08:31.976 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:08:31.975+0000 7efca86b8d80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2020-11-09T18:08:31.977 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:08:31.975+0000 7efca86b8d80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2020-11-09T18:08:31.977 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:08:31.975+0000 7efca86b8d80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2020-11-09T18:08:31.977 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:08:31.975+0000 7efca86b8d80 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2020-11-09T18:08:34.172 INFO:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2020-11-09T18:08:34.228 INFO:tasks.ceph:Reading keys from all nodes... 2020-11-09T18:08:34.229 INFO:teuthology.orchestra.run.smithi078:> set -ex 2020-11-09T18:08:34.229 INFO:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2020-11-09T18:08:34.273 INFO:teuthology.orchestra.run.smithi078.stderr:+ sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2020-11-09T18:08:34.282 INFO:teuthology.orchestra.run.smithi078.stdout:[mgr.x] 2020-11-09T18:08:34.282 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQCYhalfWalKKRAAd6XAA7tUk2SEQph1I2+zPg== 2020-11-09T18:08:34.282 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records in 2020-11-09T18:08:34.282 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records out 2020-11-09T18:08:34.282 INFO:teuthology.orchestra.run.smithi078.stderr:56 bytes copied, 0.00017725 s, 316 kB/s 2020-11-09T18:08:34.283 INFO:teuthology.orchestra.run.smithi078:> set -ex 2020-11-09T18:08:34.284 INFO:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2020-11-09T18:08:34.329 INFO:teuthology.orchestra.run.smithi078.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2020-11-09T18:08:34.337 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.0] 2020-11-09T18:08:34.337 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQCbhalf75RpFhAAAjgaspLiF/NPgAQqlyVa8Q== 2020-11-09T18:08:34.338 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records in 2020-11-09T18:08:34.338 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records out 2020-11-09T18:08:34.338 INFO:teuthology.orchestra.run.smithi078.stderr:56 bytes copied, 0.00017263 s, 324 kB/s 2020-11-09T18:08:34.339 INFO:teuthology.orchestra.run.smithi078:> set -ex 2020-11-09T18:08:34.339 INFO:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2020-11-09T18:08:34.385 INFO:teuthology.orchestra.run.smithi078.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2020-11-09T18:08:34.392 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.1] 2020-11-09T18:08:34.392 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQCdhalfEO55JxAAmE0Dm0gSroVma7LLNRR+MQ== 2020-11-09T18:08:34.393 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records in 2020-11-09T18:08:34.393 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records out 2020-11-09T18:08:34.393 INFO:teuthology.orchestra.run.smithi078.stderr:56 bytes copied, 0.00015722 s, 356 kB/s 2020-11-09T18:08:34.394 INFO:teuthology.orchestra.run.smithi078:> set -ex 2020-11-09T18:08:34.394 INFO:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2020-11-09T18:08:34.441 INFO:teuthology.orchestra.run.smithi078.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2020-11-09T18:08:34.448 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.2] 2020-11-09T18:08:34.449 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQCfhalfxgtUOhAAQOnqy0dnw5mOZPNeLxYIuw== 2020-11-09T18:08:34.449 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records in 2020-11-09T18:08:34.449 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records out 2020-11-09T18:08:34.449 INFO:teuthology.orchestra.run.smithi078.stderr:56 bytes copied, 0.000158587 s, 353 kB/s 2020-11-09T18:08:34.450 INFO:teuthology.orchestra.run.smithi078:> set -ex 2020-11-09T18:08:34.450 INFO:teuthology.orchestra.run.smithi078:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2020-11-09T18:08:34.497 INFO:teuthology.orchestra.run.smithi078.stderr:+ dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2020-11-09T18:08:34.498 INFO:teuthology.orchestra.run.smithi078.stdout:[client.0] 2020-11-09T18:08:34.498 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQCYhalfAXnMLRAAQ8BD8fplUXd6X6AS5xoZuQ== 2020-11-09T18:08:34.499 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records in 2020-11-09T18:08:34.499 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records out 2020-11-09T18:08:34.499 INFO:teuthology.orchestra.run.smithi078.stderr:59 bytes copied, 0.00015526 s, 380 kB/s 2020-11-09T18:08:34.499 INFO:teuthology.orchestra.run.smithi078:> set -ex 2020-11-09T18:08:34.500 INFO:teuthology.orchestra.run.smithi078:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2020-11-09T18:08:34.545 INFO:teuthology.orchestra.run.smithi078.stderr:+ dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2020-11-09T18:08:34.546 INFO:teuthology.orchestra.run.smithi078.stdout:[client.1] 2020-11-09T18:08:34.547 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQCYhalf5NzIMhAApBIx5b3amzLsbM0i1PyUBQ== 2020-11-09T18:08:34.547 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records in 2020-11-09T18:08:34.547 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records out 2020-11-09T18:08:34.547 INFO:teuthology.orchestra.run.smithi078.stderr:59 bytes copied, 0.000152657 s, 386 kB/s 2020-11-09T18:08:34.547 INFO:tasks.ceph:Adding keys to all mons... 2020-11-09T18:08:34.548 INFO:teuthology.orchestra.run.smithi078:> sudo tee -a /etc/ceph/c2.keyring 2020-11-09T18:08:34.602 INFO:teuthology.orchestra.run.smithi078.stdout:[mgr.x] 2020-11-09T18:08:34.602 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQCYhalfWalKKRAAd6XAA7tUk2SEQph1I2+zPg== 2020-11-09T18:08:34.602 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.0] 2020-11-09T18:08:34.602 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQCbhalf75RpFhAAAjgaspLiF/NPgAQqlyVa8Q== 2020-11-09T18:08:34.603 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.1] 2020-11-09T18:08:34.603 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQCdhalfEO55JxAAmE0Dm0gSroVma7LLNRR+MQ== 2020-11-09T18:08:34.603 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.2] 2020-11-09T18:08:34.603 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQCfhalfxgtUOhAAQOnqy0dnw5mOZPNeLxYIuw== 2020-11-09T18:08:34.603 INFO:teuthology.orchestra.run.smithi078.stdout:[client.0] 2020-11-09T18:08:34.603 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQCYhalfAXnMLRAAQ8BD8fplUXd6X6AS5xoZuQ== 2020-11-09T18:08:34.603 INFO:teuthology.orchestra.run.smithi078.stdout:[client.1] 2020-11-09T18:08:34.604 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQCYhalf5NzIMhAApBIx5b3amzLsbM0i1PyUBQ== 2020-11-09T18:08:34.604 INFO:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2020-11-09T18:08:34.687 INFO:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-09T18:08:34.774 INFO:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-09T18:08:34.859 INFO:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-09T18:08:34.948 INFO:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-11-09T18:08:35.034 INFO:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-11-09T18:08:35.075 INFO:tasks.ceph:Running mkfs on mon nodes... 2020-11-09T18:08:35.075 INFO:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2020-11-09T18:08:35.129 INFO:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2020-11-09T18:08:35.227 INFO:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2020-11-09T18:08:35.243 INFO:teuthology.orchestra.run.smithi078:> rm -- /home/ubuntu/cephtest/c2.monmap 2020-11-09T18:08:35.290 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2020-11-09T18:08:35.291 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2020-11-09T18:08:35.291 INFO:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2020-11-09T18:08:35.334 INFO:tasks.ceph.c2.mon.a:Started 2020-11-09T18:08:35.335 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2020-11-09T18:08:35.335 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2020-11-09T18:08:35.336 INFO:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2020-11-09T18:08:35.338 INFO:tasks.ceph.c2.mgr.x:Started 2020-11-09T18:08:35.338 INFO:tasks.ceph:Setting crush tunables to default 2020-11-09T18:08:35.338 INFO:teuthology.orchestra.run.smithi078:> sudo ceph --cluster c2 osd crush tunables default 2020-11-09T18:08:35.742 INFO:teuthology.orchestra.run.smithi078.stderr:adjusted tunables profile to default 2020-11-09T18:08:35.754 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2020-11-09T18:08:35.755 INFO:teuthology.orchestra.run.smithi078:> set -ex 2020-11-09T18:08:35.755 INFO:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2020-11-09T18:08:35.759 INFO:teuthology.orchestra.run.smithi078.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2020-11-09T18:08:35.768 INFO:teuthology.orchestra.run.smithi078.stdout:c4c2ebb3-c30a-46de-914d-e29a5d4f1dc8 2020-11-09T18:08:35.768 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records in 2020-11-09T18:08:35.768 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records out 2020-11-09T18:08:35.769 INFO:teuthology.orchestra.run.smithi078.stderr:37 bytes copied, 0.000126457 s, 293 kB/s 2020-11-09T18:08:35.769 INFO:teuthology.orchestra.run.smithi078:> set -ex 2020-11-09T18:08:35.770 INFO:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2020-11-09T18:08:35.817 INFO:teuthology.orchestra.run.smithi078.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2020-11-09T18:08:35.826 INFO:teuthology.orchestra.run.smithi078.stdout:1a41714f-227c-45d0-b83b-79122577f309 2020-11-09T18:08:35.826 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records in 2020-11-09T18:08:35.826 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records out 2020-11-09T18:08:35.827 INFO:teuthology.orchestra.run.smithi078.stderr:37 bytes copied, 0.000149657 s, 247 kB/s 2020-11-09T18:08:35.828 INFO:teuthology.orchestra.run.smithi078:> set -ex 2020-11-09T18:08:35.829 INFO:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2020-11-09T18:08:35.873 INFO:teuthology.orchestra.run.smithi078.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2020-11-09T18:08:35.882 INFO:teuthology.orchestra.run.smithi078.stdout:4abd7691-261e-4a2c-bc6f-57e162814a2d 2020-11-09T18:08:35.882 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records in 2020-11-09T18:08:35.882 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records out 2020-11-09T18:08:35.883 INFO:teuthology.orchestra.run.smithi078.stderr:37 bytes copied, 0.000158543 s, 233 kB/s 2020-11-09T18:08:35.883 INFO:teuthology.orchestra.run.smithi078:> sudo ceph --cluster c2 osd new c4c2ebb3-c30a-46de-914d-e29a5d4f1dc8 0 2020-11-09T18:08:36.144 INFO:teuthology.orchestra.run.smithi078.stdout:0 2020-11-09T18:08:36.156 INFO:teuthology.orchestra.run.smithi078:> sudo ceph --cluster c2 osd new 1a41714f-227c-45d0-b83b-79122577f309 1 2020-11-09T18:08:36.367 INFO:teuthology.orchestra.run.smithi078.stdout:1 2020-11-09T18:08:36.379 INFO:teuthology.orchestra.run.smithi078:> sudo ceph --cluster c2 osd new 4abd7691-261e-4a2c-bc6f-57e162814a2d 2 2020-11-09T18:08:36.601 INFO:teuthology.orchestra.run.smithi078.stdout:2 2020-11-09T18:08:36.619 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2020-11-09T18:08:36.620 INFO:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2020-11-09T18:08:36.622 INFO:tasks.ceph.c2.osd.0:Started 2020-11-09T18:08:36.622 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2020-11-09T18:08:36.622 INFO:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2020-11-09T18:08:36.624 INFO:tasks.ceph.c2.osd.1:Started 2020-11-09T18:08:36.627 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2020-11-09T18:08:36.629 INFO:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2020-11-09T18:08:36.630 INFO:tasks.ceph.c2.osd.2:Started 2020-11-09T18:08:36.631 INFO:tasks.ceph:Waiting for OSDs to come up 2020-11-09T18:08:36.934 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2020-11-09T18:08:36.994 INFO:tasks.ceph.c2.osd.1.smithi078.stderr:2020-11-09T18:08:36.995+0000 7fcb21109d80 -1 Falling back to public interface 2020-11-09T18:08:37.116 INFO:teuthology.misc.health.smithi078.stdout: 2020-11-09T18:08:37.117 INFO:teuthology.misc.health.smithi078.stdout:{"epoch":5,"fsid":"774d91bc-658c-40ed-9c19-9f7e16b77485","created":"2020-11-09T18:08:35.456646+0000","modified":"2020-11-09T18:08:36.601116+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"c4c2ebb3-c30a-46de-914d-e29a5d4f1dc8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a41714f-227c-45d0-b83b-79122577f309","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4abd7691-261e-4a2c-bc6f-57e162814a2d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-09T18:08:37.128 DEBUG:teuthology.misc:0 of 3 OSDs are up 2020-11-09T18:08:37.243 INFO:tasks.ceph.c2.osd.0.smithi078.stderr:2020-11-09T18:08:37.243+0000 7ff9004fcd80 -1 Falling back to public interface 2020-11-09T18:08:37.243 INFO:tasks.ceph.c2.osd.2.smithi078.stderr:2020-11-09T18:08:37.243+0000 7f78342d3d80 -1 Falling back to public interface 2020-11-09T18:08:37.610 INFO:tasks.ceph.c2.osd.0.smithi078.stderr:2020-11-09T18:08:37.611+0000 7ff9004fcd80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-09T18:08:37.613 INFO:tasks.ceph.c2.osd.2.smithi078.stderr:2020-11-09T18:08:37.611+0000 7f78342d3d80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-09T18:08:37.654 INFO:tasks.ceph.c2.osd.1.smithi078.stderr:2020-11-09T18:08:37.655+0000 7fcb21109d80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-09T18:08:39.482 INFO:tasks.ceph.c2.osd.2.smithi078.stderr:2020-11-09T18:08:39.483+0000 7f782b79d700 -1 osd.2 0 waiting for initial osdmap 2020-11-09T18:08:39.483 INFO:tasks.ceph.c2.osd.0.smithi078.stderr:2020-11-09T18:08:39.483+0000 7ff8f79c6700 -1 osd.0 0 waiting for initial osdmap 2020-11-09T18:08:39.483 INFO:tasks.ceph.c2.osd.1.smithi078.stderr:2020-11-09T18:08:39.483+0000 7fcb185d3700 -1 osd.1 0 waiting for initial osdmap 2020-11-09T18:08:43.431 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2020-11-09T18:08:43.710 INFO:teuthology.misc.health.smithi078.stdout: 2020-11-09T18:08:43.710 INFO:teuthology.misc.health.smithi078.stdout:{"epoch":10,"fsid":"774d91bc-658c-40ed-9c19-9f7e16b77485","created":"2020-11-09T18:08:35.456646+0000","modified":"2020-11-09T18:08:42.499646+0000","last_up_change":"2020-11-09T18:08:40.479550+0000","last_in_change":"2020-11-09T18:08:40.479550+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-09T18:08:39.265264+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"c4c2ebb3-c30a-46de-914d-e29a5d4f1dc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":14500},{"type":"v1","addr":"172.21.15.78:6810","nonce":14500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":14500},{"type":"v1","addr":"172.21.15.78:6814","nonce":14500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":14500},{"type":"v1","addr":"172.21.15.78:6822","nonce":14500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":14500},{"type":"v1","addr":"172.21.15.78:6818","nonce":14500}]},"public_addr":"172.21.15.78:6810/14500","cluster_addr":"172.21.15.78:6814/14500","heartbeat_back_addr":"172.21.15.78:6822/14500","heartbeat_front_addr":"172.21.15.78:6818/14500","state":["exists","up"]},{"osd":1,"uuid":"1a41714f-227c-45d0-b83b-79122577f309","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":8,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":14499},{"type":"v1","addr":"172.21.15.78:6801","nonce":14499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":14499},{"type":"v1","addr":"172.21.15.78:6803","nonce":14499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":14499},{"type":"v1","addr":"172.21.15.78:6807","nonce":14499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":14499},{"type":"v1","addr":"172.21.15.78:6805","nonce":14499}]},"public_addr":"172.21.15.78:6801/14499","cluster_addr":"172.21.15.78:6803/14499","heartbeat_back_addr":"172.21.15.78:6807/14499","heartbeat_front_addr":"172.21.15.78:6805/14499","state":["exists","up"]},{"osd":2,"uuid":"4abd7691-261e-4a2c-bc6f-57e162814a2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6809","nonce":14501},{"type":"v1","addr":"172.21.15.78:6811","nonce":14501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6813","nonce":14501},{"type":"v1","addr":"172.21.15.78:6815","nonce":14501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6821","nonce":14501},{"type":"v1","addr":"172.21.15.78:6823","nonce":14501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":14501},{"type":"v1","addr":"172.21.15.78:6819","nonce":14501}]},"public_addr":"172.21.15.78:6811/14501","cluster_addr":"172.21.15.78:6815/14501","heartbeat_back_addr":"172.21.15.78:6823/14501","heartbeat_front_addr":"172.21.15.78:6819/14501","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T18:08:38.627822+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T18:08:38.665862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T18:08:38.642264+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-09T18:08:43.723 DEBUG:teuthology.misc:3 of 3 OSDs are up 2020-11-09T18:08:43.723 INFO:tasks.ceph:Creating RBD pool 2020-11-09T18:08:43.723 INFO:teuthology.orchestra.run.smithi078:> sudo ceph --cluster c2 osd pool create rbd 8 2020-11-09T18:08:44.533 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'rbd' created 2020-11-09T18:08:44.546 INFO:teuthology.orchestra.run.smithi078:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2020-11-09T18:08:45.545 INFO:teuthology.orchestra.run.smithi078.stderr:enabled application 'rbd' on pool 'rbd' 2020-11-09T18:08:45.560 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2020-11-09T18:08:45.561 INFO:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2020-11-09T18:08:45.561 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2020-11-09T18:08:45.840 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:08:45.840 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":12,"fsid":"774d91bc-658c-40ed-9c19-9f7e16b77485","created":"2020-11-09T18:08:35.456646+0000","modified":"2020-11-09T18:08:45.523495+0000","last_up_change":"2020-11-09T18:08:40.479550+0000","last_in_change":"2020-11-09T18:08:40.479550+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-09T18:08:39.265264+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-09T18:08:44.019222+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"c4c2ebb3-c30a-46de-914d-e29a5d4f1dc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":14500},{"type":"v1","addr":"172.21.15.78:6810","nonce":14500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":14500},{"type":"v1","addr":"172.21.15.78:6814","nonce":14500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":14500},{"type":"v1","addr":"172.21.15.78:6822","nonce":14500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":14500},{"type":"v1","addr":"172.21.15.78:6818","nonce":14500}]},"public_addr":"172.21.15.78:6810/14500","cluster_addr":"172.21.15.78:6814/14500","heartbeat_back_addr":"172.21.15.78:6822/14500","heartbeat_front_addr":"172.21.15.78:6818/14500","state":["exists","up"]},{"osd":1,"uuid":"1a41714f-227c-45d0-b83b-79122577f309","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":14499},{"type":"v1","addr":"172.21.15.78:6801","nonce":14499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":14499},{"type":"v1","addr":"172.21.15.78:6803","nonce":14499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":14499},{"type":"v1","addr":"172.21.15.78:6807","nonce":14499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":14499},{"type":"v1","addr":"172.21.15.78:6805","nonce":14499}]},"public_addr":"172.21.15.78:6801/14499","cluster_addr":"172.21.15.78:6803/14499","heartbeat_back_addr":"172.21.15.78:6807/14499","heartbeat_front_addr":"172.21.15.78:6805/14499","state":["exists","up"]},{"osd":2,"uuid":"4abd7691-261e-4a2c-bc6f-57e162814a2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6809","nonce":14501},{"type":"v1","addr":"172.21.15.78:6811","nonce":14501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6813","nonce":14501},{"type":"v1","addr":"172.21.15.78:6815","nonce":14501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6821","nonce":14501},{"type":"v1","addr":"172.21.15.78:6823","nonce":14501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":14501},{"type":"v1","addr":"172.21.15.78:6819","nonce":14501}]},"public_addr":"172.21.15.78:6811/14501","cluster_addr":"172.21.15.78:6815/14501","heartbeat_back_addr":"172.21.15.78:6823/14501","heartbeat_front_addr":"172.21.15.78:6819/14501","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T18:08:38.627822+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T18:08:38.665862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T18:08:38.642264+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-09T18:08:45.854 INFO:tasks.ceph.ceph_manager.c2:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-09T18:08:39.265264+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '10', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-09T18:08:44.019222+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-11-09T18:08:45.855 INFO:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd pool get device_health_metrics pg_num 2020-11-09T18:08:46.147 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2020-11-09T18:08:46.160 INFO:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd pool get rbd pg_num 2020-11-09T18:08:46.452 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 8 2020-11-09T18:08:46.466 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2020-11-09T18:08:46.466 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2020-11-09T18:08:46.467 INFO:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2020-11-09T18:08:46.784 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:08:46.798 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":4,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":14316},{"type":"v1","addr":"172.21.15.78:6825","nonce":14316}]},"active_addr":"172.21.15.78:6825/14316","active_change":"2020-11-09T18:08:39.237986+0000","active_mgr_features":4540138297136906239,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.6.2","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":316073635}]},{"addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":1102053254}]},{"addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":1280914763}]}]}} 2020-11-09T18:08:46.800 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2020-11-09T18:08:46.800 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2020-11-09T18:08:46.801 INFO:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2020-11-09T18:08:47.091 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:08:47.092 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":13,"fsid":"774d91bc-658c-40ed-9c19-9f7e16b77485","created":"2020-11-09T18:08:35.456646+0000","modified":"2020-11-09T18:08:46.533139+0000","last_up_change":"2020-11-09T18:08:40.479550+0000","last_in_change":"2020-11-09T18:08:40.479550+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-09T18:08:39.265264+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-09T18:08:44.019222+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"c4c2ebb3-c30a-46de-914d-e29a5d4f1dc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":14500},{"type":"v1","addr":"172.21.15.78:6810","nonce":14500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":14500},{"type":"v1","addr":"172.21.15.78:6814","nonce":14500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":14500},{"type":"v1","addr":"172.21.15.78:6822","nonce":14500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":14500},{"type":"v1","addr":"172.21.15.78:6818","nonce":14500}]},"public_addr":"172.21.15.78:6810/14500","cluster_addr":"172.21.15.78:6814/14500","heartbeat_back_addr":"172.21.15.78:6822/14500","heartbeat_front_addr":"172.21.15.78:6818/14500","state":["exists","up"]},{"osd":1,"uuid":"1a41714f-227c-45d0-b83b-79122577f309","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":14499},{"type":"v1","addr":"172.21.15.78:6801","nonce":14499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":14499},{"type":"v1","addr":"172.21.15.78:6803","nonce":14499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":14499},{"type":"v1","addr":"172.21.15.78:6807","nonce":14499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":14499},{"type":"v1","addr":"172.21.15.78:6805","nonce":14499}]},"public_addr":"172.21.15.78:6801/14499","cluster_addr":"172.21.15.78:6803/14499","heartbeat_back_addr":"172.21.15.78:6807/14499","heartbeat_front_addr":"172.21.15.78:6805/14499","state":["exists","up"]},{"osd":2,"uuid":"4abd7691-261e-4a2c-bc6f-57e162814a2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6809","nonce":14501},{"type":"v1","addr":"172.21.15.78:6811","nonce":14501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6813","nonce":14501},{"type":"v1","addr":"172.21.15.78:6815","nonce":14501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6821","nonce":14501},{"type":"v1","addr":"172.21.15.78:6823","nonce":14501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":14501},{"type":"v1","addr":"172.21.15.78:6819","nonce":14501}]},"public_addr":"172.21.15.78:6811/14501","cluster_addr":"172.21.15.78:6815/14501","heartbeat_back_addr":"172.21.15.78:6823/14501","heartbeat_front_addr":"172.21.15.78:6819/14501","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T18:08:38.627822+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T18:08:38.665862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T18:08:38.642264+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-09T18:08:47.106 INFO:tasks.ceph.ceph_manager.c2:all up! 2020-11-09T18:08:47.106 INFO:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2020-11-09T18:08:47.393 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:08:47.393 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":13,"fsid":"774d91bc-658c-40ed-9c19-9f7e16b77485","created":"2020-11-09T18:08:35.456646+0000","modified":"2020-11-09T18:08:46.533139+0000","last_up_change":"2020-11-09T18:08:40.479550+0000","last_in_change":"2020-11-09T18:08:40.479550+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-09T18:08:39.265264+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-09T18:08:44.019222+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"c4c2ebb3-c30a-46de-914d-e29a5d4f1dc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":14500},{"type":"v1","addr":"172.21.15.78:6810","nonce":14500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":14500},{"type":"v1","addr":"172.21.15.78:6814","nonce":14500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":14500},{"type":"v1","addr":"172.21.15.78:6822","nonce":14500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":14500},{"type":"v1","addr":"172.21.15.78:6818","nonce":14500}]},"public_addr":"172.21.15.78:6810/14500","cluster_addr":"172.21.15.78:6814/14500","heartbeat_back_addr":"172.21.15.78:6822/14500","heartbeat_front_addr":"172.21.15.78:6818/14500","state":["exists","up"]},{"osd":1,"uuid":"1a41714f-227c-45d0-b83b-79122577f309","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":14499},{"type":"v1","addr":"172.21.15.78:6801","nonce":14499}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":14499},{"type":"v1","addr":"172.21.15.78:6803","nonce":14499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":14499},{"type":"v1","addr":"172.21.15.78:6807","nonce":14499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":14499},{"type":"v1","addr":"172.21.15.78:6805","nonce":14499}]},"public_addr":"172.21.15.78:6801/14499","cluster_addr":"172.21.15.78:6803/14499","heartbeat_back_addr":"172.21.15.78:6807/14499","heartbeat_front_addr":"172.21.15.78:6805/14499","state":["exists","up"]},{"osd":2,"uuid":"4abd7691-261e-4a2c-bc6f-57e162814a2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6809","nonce":14501},{"type":"v1","addr":"172.21.15.78:6811","nonce":14501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6813","nonce":14501},{"type":"v1","addr":"172.21.15.78:6815","nonce":14501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6821","nonce":14501},{"type":"v1","addr":"172.21.15.78:6823","nonce":14501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":14501},{"type":"v1","addr":"172.21.15.78:6819","nonce":14501}]},"public_addr":"172.21.15.78:6811/14501","cluster_addr":"172.21.15.78:6815/14501","heartbeat_back_addr":"172.21.15.78:6823/14501","heartbeat_front_addr":"172.21.15.78:6819/14501","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T18:08:38.627822+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T18:08:38.665862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T18:08:38.642264+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-09T18:08:47.407 INFO:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2020-11-09T18:08:47.575 INFO:teuthology.orchestra.run.smithi078.stdout:34359738372 2020-11-09T18:08:47.576 INFO:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2020-11-09T18:08:47.745 INFO:teuthology.orchestra.run.smithi078.stdout:34359738371 2020-11-09T18:08:47.745 INFO:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2020-11-09T18:08:47.913 INFO:teuthology.orchestra.run.smithi078.stdout:34359738372 2020-11-09T18:08:47.914 INFO:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2020-11-09T18:08:48.217 INFO:teuthology.orchestra.run.smithi078.stdout:34359738371 2020-11-09T18:08:48.231 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738371 for osd.0 2020-11-09T18:08:49.233 INFO:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2020-11-09T18:08:49.538 INFO:teuthology.orchestra.run.smithi078.stdout:34359738371 2020-11-09T18:08:49.555 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738371 for osd.0 2020-11-09T18:08:50.557 INFO:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2020-11-09T18:08:50.857 INFO:teuthology.orchestra.run.smithi078.stdout:34359738372 2020-11-09T18:08:50.871 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738372 for osd.0 2020-11-09T18:08:50.872 INFO:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2020-11-09T18:08:51.166 INFO:teuthology.orchestra.run.smithi078.stdout:34359738371 2020-11-09T18:08:51.179 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.1 2020-11-09T18:08:51.180 INFO:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2020-11-09T18:08:51.466 INFO:teuthology.orchestra.run.smithi078.stdout:34359738372 2020-11-09T18:08:51.481 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738372 for osd.2 2020-11-09T18:08:51.481 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2020-11-09T18:08:51.482 INFO:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2020-11-09T18:08:51.772 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:08:51.774 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2020-11-09T18:08:51.788 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2020-11-09T18:08:51.248499+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14696,"kb_used_data":336,"kb_used_omap":0,"kb_used_meta":14336,"kb_avail":314558104,"statfs":{"total":322122547200,"available":322107498496,"internally_reserved":0,"allocated":344064,"data_stored":99408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14680064},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.710698"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T18:08:45.550401+0000","last_change":"2020-11-09T18:08:45.548992+0000","last_active":"2020-11-09T18:08:45.550401+0000","last_peered":"2020-11-09T18:08:45.550401+0000","last_clean":"2020-11-09T18:08:45.550401+0000","last_became_active":"2020-11-09T18:08:45.548600+0000","last_became_peered":"2020-11-09T18:08:45.548600+0000","last_unstale":"2020-11-09T18:08:45.550401+0000","last_undegraded":"2020-11-09T18:08:45.550401+0000","last_fullsized":"2020-11-09T18:08:45.550401+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T18:08:44.514798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T18:08:44.514798+0000","last_clean_scrub_stamp":"2020-11-09T18:08:44.514798+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T18:08:45.549703+0000","last_change":"2020-11-09T18:08:45.548548+0000","last_active":"2020-11-09T18:08:45.549703+0000","last_peered":"2020-11-09T18:08:45.549703+0000","last_clean":"2020-11-09T18:08:45.549703+0000","last_became_active":"2020-11-09T18:08:45.548115+0000","last_became_peered":"2020-11-09T18:08:45.548115+0000","last_unstale":"2020-11-09T18:08:45.549703+0000","last_undegraded":"2020-11-09T18:08:45.549703+0000","last_fullsized":"2020-11-09T18:08:45.549703+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T18:08:44.514798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T18:08:44.514798+0000","last_clean_scrub_stamp":"2020-11-09T18:08:44.514798+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T18:08:45.552931+0000","last_change":"2020-11-09T18:08:45.551562+0000","last_active":"2020-11-09T18:08:45.552931+0000","last_peered":"2020-11-09T18:08:45.552931+0000","last_clean":"2020-11-09T18:08:45.552931+0000","last_became_active":"2020-11-09T18:08:45.551189+0000","last_became_peered":"2020-11-09T18:08:45.551189+0000","last_unstale":"2020-11-09T18:08:45.552931+0000","last_undegraded":"2020-11-09T18:08:45.552931+0000","last_fullsized":"2020-11-09T18:08:45.552931+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T18:08:44.514798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T18:08:44.514798+0000","last_clean_scrub_stamp":"2020-11-09T18:08:44.514798+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T18:08:45.552058+0000","last_change":"2020-11-09T18:08:45.547005+0000","last_active":"2020-11-09T18:08:45.552058+0000","last_peered":"2020-11-09T18:08:45.552058+0000","last_clean":"2020-11-09T18:08:45.552058+0000","last_became_active":"2020-11-09T18:08:45.546510+0000","last_became_peered":"2020-11-09T18:08:45.546510+0000","last_unstale":"2020-11-09T18:08:45.552058+0000","last_undegraded":"2020-11-09T18:08:45.552058+0000","last_fullsized":"2020-11-09T18:08:45.552058+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T18:08:44.514798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T18:08:44.514798+0000","last_clean_scrub_stamp":"2020-11-09T18:08:44.514798+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"12","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T18:08:45.543402+0000","last_change":"2020-11-09T18:08:41.538834+0000","last_active":"2020-11-09T18:08:45.543402+0000","last_peered":"2020-11-09T18:08:45.543402+0000","last_clean":"2020-11-09T18:08:45.543402+0000","last_became_active":"2020-11-09T18:08:41.537296+0000","last_became_peered":"2020-11-09T18:08:41.537296+0000","last_unstale":"2020-11-09T18:08:45.543402+0000","last_undegraded":"2020-11-09T18:08:45.543402+0000","last_fullsized":"2020-11-09T18:08:45.543402+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T18:08:39.470460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T18:08:39.470460+0000","last_clean_scrub_stamp":"2020-11-09T18:08:39.470460+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T18:08:45.553099+0000","last_change":"2020-11-09T18:08:45.551886+0000","last_active":"2020-11-09T18:08:45.553099+0000","last_peered":"2020-11-09T18:08:45.553099+0000","last_clean":"2020-11-09T18:08:45.553099+0000","last_became_active":"2020-11-09T18:08:45.550268+0000","last_became_peered":"2020-11-09T18:08:45.550268+0000","last_unstale":"2020-11-09T18:08:45.553099+0000","last_undegraded":"2020-11-09T18:08:45.553099+0000","last_fullsized":"2020-11-09T18:08:45.553099+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T18:08:44.514798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T18:08:44.514798+0000","last_clean_scrub_stamp":"2020-11-09T18:08:44.514798+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T18:08:45.551856+0000","last_change":"2020-11-09T18:08:45.549399+0000","last_active":"2020-11-09T18:08:45.551856+0000","last_peered":"2020-11-09T18:08:45.551856+0000","last_clean":"2020-11-09T18:08:45.551856+0000","last_became_active":"2020-11-09T18:08:45.549001+0000","last_became_peered":"2020-11-09T18:08:45.549001+0000","last_unstale":"2020-11-09T18:08:45.551856+0000","last_undegraded":"2020-11-09T18:08:45.551856+0000","last_fullsized":"2020-11-09T18:08:45.551856+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T18:08:44.514798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T18:08:44.514798+0000","last_clean_scrub_stamp":"2020-11-09T18:08:44.514798+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T18:08:45.549519+0000","last_change":"2020-11-09T18:08:45.548393+0000","last_active":"2020-11-09T18:08:45.549519+0000","last_peered":"2020-11-09T18:08:45.549519+0000","last_clean":"2020-11-09T18:08:45.549519+0000","last_became_active":"2020-11-09T18:08:45.548037+0000","last_became_peered":"2020-11-09T18:08:45.548037+0000","last_unstale":"2020-11-09T18:08:45.549519+0000","last_undegraded":"2020-11-09T18:08:45.549519+0000","last_fullsized":"2020-11-09T18:08:45.549519+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T18:08:44.514798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T18:08:44.514798+0000","last_clean_scrub_stamp":"2020-11-09T18:08:44.514798+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T18:08:45.550683+0000","last_change":"2020-11-09T18:08:45.549306+0000","last_active":"2020-11-09T18:08:45.550683+0000","last_peered":"2020-11-09T18:08:45.550683+0000","last_clean":"2020-11-09T18:08:45.550683+0000","last_became_active":"2020-11-09T18:08:45.548852+0000","last_became_peered":"2020-11-09T18:08:45.548852+0000","last_unstale":"2020-11-09T18:08:45.550683+0000","last_undegraded":"2020-11-09T18:08:45.550683+0000","last_fullsized":"2020-11-09T18:08:45.550683+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T18:08:44.514798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T18:08:44.514798+0000","last_clean_scrub_stamp":"2020-11-09T18:08:44.514798+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4856,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":104852744,"statfs":{"total":107374182400,"available":107369209856,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.147}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91700000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2649999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92000000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.181}]}]}]}} 2020-11-09T18:08:51.790 INFO:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2020-11-09T18:08:52.082 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:08:52.082 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2020-11-09T18:08:52.090 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2020-11-09T18:08:51.248499+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14696,"kb_used_data":336,"kb_used_omap":0,"kb_used_meta":14336,"kb_avail":314558104,"statfs":{"total":322122547200,"available":322107498496,"internally_reserved":0,"allocated":344064,"data_stored":99408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14680064},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.710698"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T18:08:45.550401+0000","last_change":"2020-11-09T18:08:45.548992+0000","last_active":"2020-11-09T18:08:45.550401+0000","last_peered":"2020-11-09T18:08:45.550401+0000","last_clean":"2020-11-09T18:08:45.550401+0000","last_became_active":"2020-11-09T18:08:45.548600+0000","last_became_peered":"2020-11-09T18:08:45.548600+0000","last_unstale":"2020-11-09T18:08:45.550401+0000","last_undegraded":"2020-11-09T18:08:45.550401+0000","last_fullsized":"2020-11-09T18:08:45.550401+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T18:08:44.514798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T18:08:44.514798+0000","last_clean_scrub_stamp":"2020-11-09T18:08:44.514798+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T18:08:45.549703+0000","last_change":"2020-11-09T18:08:45.548548+0000","last_active":"2020-11-09T18:08:45.549703+0000","last_peered":"2020-11-09T18:08:45.549703+0000","last_clean":"2020-11-09T18:08:45.549703+0000","last_became_active":"2020-11-09T18:08:45.548115+0000","last_became_peered":"2020-11-09T18:08:45.548115+0000","last_unstale":"2020-11-09T18:08:45.549703+0000","last_undegraded":"2020-11-09T18:08:45.549703+0000","last_fullsized":"2020-11-09T18:08:45.549703+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T18:08:44.514798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T18:08:44.514798+0000","last_clean_scrub_stamp":"2020-11-09T18:08:44.514798+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T18:08:45.552931+0000","last_change":"2020-11-09T18:08:45.551562+0000","last_active":"2020-11-09T18:08:45.552931+0000","last_peered":"2020-11-09T18:08:45.552931+0000","last_clean":"2020-11-09T18:08:45.552931+0000","last_became_active":"2020-11-09T18:08:45.551189+0000","last_became_peered":"2020-11-09T18:08:45.551189+0000","last_unstale":"2020-11-09T18:08:45.552931+0000","last_undegraded":"2020-11-09T18:08:45.552931+0000","last_fullsized":"2020-11-09T18:08:45.552931+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T18:08:44.514798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T18:08:44.514798+0000","last_clean_scrub_stamp":"2020-11-09T18:08:44.514798+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T18:08:45.552058+0000","last_change":"2020-11-09T18:08:45.547005+0000","last_active":"2020-11-09T18:08:45.552058+0000","last_peered":"2020-11-09T18:08:45.552058+0000","last_clean":"2020-11-09T18:08:45.552058+0000","last_became_active":"2020-11-09T18:08:45.546510+0000","last_became_peered":"2020-11-09T18:08:45.546510+0000","last_unstale":"2020-11-09T18:08:45.552058+0000","last_undegraded":"2020-11-09T18:08:45.552058+0000","last_fullsized":"2020-11-09T18:08:45.552058+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T18:08:44.514798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T18:08:44.514798+0000","last_clean_scrub_stamp":"2020-11-09T18:08:44.514798+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"12","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T18:08:45.543402+0000","last_change":"2020-11-09T18:08:41.538834+0000","last_active":"2020-11-09T18:08:45.543402+0000","last_peered":"2020-11-09T18:08:45.543402+0000","last_clean":"2020-11-09T18:08:45.543402+0000","last_became_active":"2020-11-09T18:08:41.537296+0000","last_became_peered":"2020-11-09T18:08:41.537296+0000","last_unstale":"2020-11-09T18:08:45.543402+0000","last_undegraded":"2020-11-09T18:08:45.543402+0000","last_fullsized":"2020-11-09T18:08:45.543402+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T18:08:39.470460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T18:08:39.470460+0000","last_clean_scrub_stamp":"2020-11-09T18:08:39.470460+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T18:08:45.553099+0000","last_change":"2020-11-09T18:08:45.551886+0000","last_active":"2020-11-09T18:08:45.553099+0000","last_peered":"2020-11-09T18:08:45.553099+0000","last_clean":"2020-11-09T18:08:45.553099+0000","last_became_active":"2020-11-09T18:08:45.550268+0000","last_became_peered":"2020-11-09T18:08:45.550268+0000","last_unstale":"2020-11-09T18:08:45.553099+0000","last_undegraded":"2020-11-09T18:08:45.553099+0000","last_fullsized":"2020-11-09T18:08:45.553099+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T18:08:44.514798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T18:08:44.514798+0000","last_clean_scrub_stamp":"2020-11-09T18:08:44.514798+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T18:08:45.551856+0000","last_change":"2020-11-09T18:08:45.549399+0000","last_active":"2020-11-09T18:08:45.551856+0000","last_peered":"2020-11-09T18:08:45.551856+0000","last_clean":"2020-11-09T18:08:45.551856+0000","last_became_active":"2020-11-09T18:08:45.549001+0000","last_became_peered":"2020-11-09T18:08:45.549001+0000","last_unstale":"2020-11-09T18:08:45.551856+0000","last_undegraded":"2020-11-09T18:08:45.551856+0000","last_fullsized":"2020-11-09T18:08:45.551856+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T18:08:44.514798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T18:08:44.514798+0000","last_clean_scrub_stamp":"2020-11-09T18:08:44.514798+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T18:08:45.549519+0000","last_change":"2020-11-09T18:08:45.548393+0000","last_active":"2020-11-09T18:08:45.549519+0000","last_peered":"2020-11-09T18:08:45.549519+0000","last_clean":"2020-11-09T18:08:45.549519+0000","last_became_active":"2020-11-09T18:08:45.548037+0000","last_became_peered":"2020-11-09T18:08:45.548037+0000","last_unstale":"2020-11-09T18:08:45.549519+0000","last_undegraded":"2020-11-09T18:08:45.549519+0000","last_fullsized":"2020-11-09T18:08:45.549519+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T18:08:44.514798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T18:08:44.514798+0000","last_clean_scrub_stamp":"2020-11-09T18:08:44.514798+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T18:08:45.550683+0000","last_change":"2020-11-09T18:08:45.549306+0000","last_active":"2020-11-09T18:08:45.550683+0000","last_peered":"2020-11-09T18:08:45.550683+0000","last_clean":"2020-11-09T18:08:45.550683+0000","last_became_active":"2020-11-09T18:08:45.548852+0000","last_became_peered":"2020-11-09T18:08:45.548852+0000","last_unstale":"2020-11-09T18:08:45.550683+0000","last_undegraded":"2020-11-09T18:08:45.550683+0000","last_fullsized":"2020-11-09T18:08:45.550683+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T18:08:44.514798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T18:08:44.514798+0000","last_clean_scrub_stamp":"2020-11-09T18:08:44.514798+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4856,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":104852744,"statfs":{"total":107374182400,"available":107369209856,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.147}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91700000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2649999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92000000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.181}]}]}]}} 2020-11-09T18:08:52.091 INFO:tasks.ceph.ceph_manager.c2:clean! 2020-11-09T18:08:52.092 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2020-11-09T18:08:52.092 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2020-11-09T18:08:52.092 INFO:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2020-11-09T18:08:52.451 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:08:52.465 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2020-11-09T18:08:52.466 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2020-11-09T18:08:52.466 INFO:teuthology.run_tasks:Running task rgw... 2020-11-09T18:08:52.490 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2020-11-09T18:08:52.491 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2020-11-09T18:08:52.491 INFO:tasks.rgw:Creating data pools 2020-11-09T18:08:52.492 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2020-11-09T18:08:52.492 INFO:teuthology.orchestra.run.smithi051:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2020-11-09T18:08:53.476 INFO:teuthology.orchestra.run.smithi051.stderr:pool 'default.rgw.buckets.data' created 2020-11-09T18:08:53.494 INFO:teuthology.orchestra.run.smithi051:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2020-11-09T18:08:54.491 INFO:teuthology.orchestra.run.smithi051.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-11-09T18:08:54.506 INFO:teuthology.orchestra.run.smithi051:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2020-11-09T18:08:55.500 INFO:teuthology.orchestra.run.smithi051.stderr:pool 'default.rgw.buckets.index' created 2020-11-09T18:08:55.516 INFO:teuthology.orchestra.run.smithi051:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2020-11-09T18:08:56.503 INFO:teuthology.orchestra.run.smithi051.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-11-09T18:08:56.529 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2020-11-09T18:08:56.529 INFO:teuthology.orchestra.run.smithi051:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2020-11-09T18:08:56.823 INFO:teuthology.orchestra.run.smithi051.stderr:pool 'default.rgw.buckets.data' already exists 2020-11-09T18:08:56.834 INFO:teuthology.orchestra.run.smithi051:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2020-11-09T18:08:57.504 INFO:teuthology.orchestra.run.smithi051.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-11-09T18:08:57.518 INFO:teuthology.orchestra.run.smithi051:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2020-11-09T18:08:57.815 INFO:teuthology.orchestra.run.smithi051.stderr:pool 'default.rgw.buckets.index' already exists 2020-11-09T18:08:57.834 INFO:teuthology.orchestra.run.smithi051:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2020-11-09T18:08:58.508 INFO:teuthology.orchestra.run.smithi051.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-11-09T18:08:58.522 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2020-11-09T18:08:58.522 INFO:teuthology.orchestra.run.smithi078:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2020-11-09T18:08:59.468 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'default.rgw.buckets.data' created 2020-11-09T18:08:59.489 INFO:teuthology.orchestra.run.smithi078:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2020-11-09T18:09:00.482 INFO:teuthology.orchestra.run.smithi078.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-11-09T18:09:00.504 INFO:teuthology.orchestra.run.smithi078:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2020-11-09T18:09:01.492 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'default.rgw.buckets.index' created 2020-11-09T18:09:01.509 INFO:teuthology.orchestra.run.smithi078:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2020-11-09T18:09:02.513 INFO:teuthology.orchestra.run.smithi078.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-11-09T18:09:02.537 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2020-11-09T18:09:02.538 INFO:teuthology.orchestra.run.smithi078:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2020-11-09T18:09:02.809 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'default.rgw.buckets.data' already exists 2020-11-09T18:09:02.821 INFO:teuthology.orchestra.run.smithi078:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2020-11-09T18:09:03.497 INFO:teuthology.orchestra.run.smithi078.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-11-09T18:09:03.511 INFO:teuthology.orchestra.run.smithi078:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2020-11-09T18:09:03.807 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'default.rgw.buckets.index' already exists 2020-11-09T18:09:03.818 INFO:teuthology.orchestra.run.smithi078:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2020-11-09T18:09:04.497 INFO:teuthology.orchestra.run.smithi078.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-11-09T18:09:04.511 DEBUG:tasks.rgw:Pools created 2020-11-09T18:09:04.511 INFO:tasks.rgw:Configuring compression type = random 2020-11-09T18:09:04.512 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2020-11-09T18:09:04.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2020-11-09T18:09:04.513 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2020-11-09T18:09:04.549 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:09:04.549 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:09:04.561 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:04.561+0000 7f6526a8d0c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:06.446 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:06.445+0000 7f6526a8d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:06.446 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:06.445+0000 7f6526a8d0c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:06.452 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:06.449+0000 7f6526a8d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:06.453 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:06.449+0000 7f6526a8d0c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:06.454 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:06.449+0000 7f6526a8d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:06.454 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:06.449+0000 7f6526a8d0c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:06.455 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:06.453+0000 7f6526a8d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:06.477 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:06.473+0000 7f6526a8d0c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:06.478 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:06.473+0000 7f6526a8d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:06.478 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:06.473+0000 7f6526a8d0c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:06.478 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:06.473+0000 7f6526a8d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:06.478 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:06.473+0000 7f6526a8d0c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:06.478 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:06.473+0000 7f6526a8d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:06.479 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:06.473+0000 7f6526a8d0c0 10 cannot find current period zonegroup using local zonegroup 2020-11-09T18:09:06.479 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:06.473+0000 7f6526a8d0c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:06.479 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:06.477+0000 7f6526a8d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:06.479 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:06.477+0000 7f6526a8d0c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:06.479 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:06.477+0000 7f6526a8d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:06.479 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:06.477+0000 7f6526a8d0c0 10 Creating default zonegroup 2020-11-09T18:09:06.479 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:06.477+0000 7f6526a8d0c0 20 get_system_obj_state: rctx=0x7ffeeb64be30 obj=:.pools.avail state=0x5559339762e0 s->prefetch_data=0 2020-11-09T18:09:06.480 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:06.477+0000 7f6526a8d0c0 10 couldn't find old data placement pools config, setting up new ones for the zone 2020-11-09T18:09:06.494 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:06.493+0000 7f6526a8d0c0 10 WARNING: store->list_zones() returned r=-2 2020-11-09T18:09:06.495 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:06.493+0000 7f6526a8d0c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:06.495 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:06.493+0000 7f6526a8d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:06.500 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:06.497+0000 7f6526a8d0c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:06.500 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:06.497+0000 7f6526a8d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:06.501 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:06.497+0000 7f6526a8d0c0 10 could not read realm id: (2) No such file or directory 2020-11-09T18:09:06.501 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:06.497+0000 7f6526a8d0c0 10 WARNING: failed to set zone as default, r=-22 2020-11-09T18:09:06.501 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:06.497+0000 7f6526a8d0c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:06.501 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:06.501+0000 7f6526a8d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:06.509 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:06.509+0000 7f6526a8d0c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:06.510 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:06.509+0000 7f6526a8d0c0 20 rados_obj.operate() r=0 bl.length=873 2020-11-09T18:09:06.510 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:06.509+0000 7f6526a8d0c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:06.511 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:06.509+0000 7f6526a8d0c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-09T18:09:06.511 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:06.509+0000 7f6526a8d0c0 20 zonegroup default 2020-11-09T18:09:06.512 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:06.509+0000 7f6526a8d0c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:06.513 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:06.509+0000 7f6526a8d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:06.514 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:06.509+0000 7f6526a8d0c0 10 Cannot find current period zone using local zone 2020-11-09T18:09:06.514 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:06.509+0000 7f6526a8d0c0 10 Using default name default 2020-11-09T18:09:06.514 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:06.509+0000 7f6526a8d0c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:06.514 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:06.509+0000 7f6526a8d0c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-09T18:09:06.514 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:06.509+0000 7f6526a8d0c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:06.515 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:06.513+0000 7f6526a8d0c0 20 rados_obj.operate() r=0 bl.length=873 2020-11-09T18:09:06.515 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:06.513+0000 7f6526a8d0c0 20 zone default found 2020-11-09T18:09:06.515 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:06.513+0000 7f6526a8d0c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:08.483 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:08.481+0000 7f6526a8d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:08.484 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:08.481+0000 7f6526a8d0c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:08.485 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:08.481+0000 7f6526a8d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:08.485 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:08.481+0000 7f6526a8d0c0 20 started sync module instance, tier type = 2020-11-09T18:09:08.485 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:08.481+0000 7f6526a8d0c0 20 started zone id=90fe8498-2ca2-45cd-aec4-98a2c19bc03a (name=default) with tier type = 2020-11-09T18:09:10.530 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:10.529+0000 7f6526a8d0c0 20 add_watcher() i=0 2020-11-09T18:09:10.531 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:10.529+0000 7f6526a8d0c0 20 add_watcher() i=1 2020-11-09T18:09:10.531 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:10.529+0000 7f6526a8d0c0 20 add_watcher() i=2 2020-11-09T18:09:10.531 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:10.529+0000 7f6526a8d0c0 20 add_watcher() i=3 2020-11-09T18:09:10.531 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:10.529+0000 7f6526a8d0c0 20 add_watcher() i=4 2020-11-09T18:09:10.531 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:10.529+0000 7f6526a8d0c0 20 add_watcher() i=5 2020-11-09T18:09:10.531 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:10.529+0000 7f6526a8d0c0 20 add_watcher() i=6 2020-11-09T18:09:10.532 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:10.529+0000 7f6526a8d0c0 20 add_watcher() i=7 2020-11-09T18:09:10.532 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:10.529+0000 7f6526a8d0c0 2 all 8 watchers are set, enabling cache 2020-11-09T18:09:10.533 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:10.533+0000 7f6526a8d0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T18:09:10.534 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:10.533+0000 7f6526a8d0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T18:09:10.534 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:10.533+0000 7f6526a8d0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T18:09:10.537 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:10.533+0000 7f6526a8d0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T18:09:10.538 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:10.533+0000 7f6526a8d0c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-09T18:09:10.538 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:10.537+0000 7f6526a8d0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T18:09:10.539 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:10.537+0000 7f6526a8d0c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-09T18:09:10.541 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:10.541+0000 7f6526a8d0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T18:09:10.542 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:10.541+0000 7f6526a8d0c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-09T18:09:10.545 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:10.545+0000 7f6526a8d0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T18:09:10.546 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:10.545+0000 7f6526a8d0c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-09T18:09:10.549 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:10.549+0000 7f6526a8d0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T18:09:10.551 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:10.549+0000 7f64e2fed700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-09T18:09:10.551 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:10.549+0000 7f6526a8d0c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-09T18:09:10.551 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:10.549+0000 7f6526a8d0c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-09T18:09:14.507 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.505+0000 7f6526a8d0c0 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2020-11-09T18:09:14.642 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.641+0000 7f6451ffb700 20 reqs_thread_entry: start 2020-11-09T18:09:14.642 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.641+0000 7f6526a8d0c0 20 init_complete bucket index max shards: 11 2020-11-09T18:09:14.642 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.641+0000 7f64377fe700 20 reqs_thread_entry: start 2020-11-09T18:09:14.643 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.641+0000 7f6526a8d0c0 10 Started notification manager with: 1 workers 2020-11-09T18:09:14.643 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.641+0000 7f6435ffb700 20 INFO: next queues processing will happen at: Mon Nov 9 18:09:45 2020 2020-11-09T18:09:14.643 INFO:teuthology.orchestra.run.smithi051.stderr: 2020-11-09T18:09:14.643 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.641+0000 7f64357fa700 20 reqs_thread_entry: start 2020-11-09T18:09:14.670 INFO:teuthology.orchestra.run.smithi051.stdout:[] 2020-11-09T18:09:14.670 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.669+0000 7f6526a8d0c0 20 remove_watcher() i=0 2020-11-09T18:09:14.671 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.669+0000 7f6526a8d0c0 2 removed watcher, disabling cache 2020-11-09T18:09:14.673 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.673+0000 7f6526a8d0c0 20 remove_watcher() i=1 2020-11-09T18:09:14.676 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.673+0000 7f6526a8d0c0 20 remove_watcher() i=2 2020-11-09T18:09:14.679 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.677+0000 7f6526a8d0c0 20 remove_watcher() i=3 2020-11-09T18:09:14.682 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.681+0000 7f6526a8d0c0 20 remove_watcher() i=4 2020-11-09T18:09:14.685 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.685+0000 7f6526a8d0c0 20 remove_watcher() i=5 2020-11-09T18:09:14.688 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.685+0000 7f6526a8d0c0 20 remove_watcher() i=6 2020-11-09T18:09:14.693 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.693+0000 7f6526a8d0c0 20 remove_watcher() i=7 2020-11-09T18:09:14.695 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.693+0000 7f64e2fed700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-09T18:09:14.712 DEBUG:tasks.util.rgw: json result: [] 2020-11-09T18:09:14.712 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-09T18:09:14.712 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-09T18:09:14.713 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-11-09T18:09:14.795 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:09:14.795 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:09:14.810 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.809+0000 7f7b083250c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-09T18:09:14.810 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.809+0000 7f7b083250c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-09T18:09:14.816 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.813+0000 7f7a37fff700 20 reqs_thread_entry: start 2020-11-09T18:09:14.817 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.813+0000 7f7b083250c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:14.818 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.817+0000 7f7b083250c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-09T18:09:14.818 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.817+0000 7f7b083250c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:14.819 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.817+0000 7f7b083250c0 20 rados_obj.operate() r=0 bl.length=873 2020-11-09T18:09:14.819 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.817+0000 7f7b083250c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:14.820 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.817+0000 7f7b083250c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:14.820 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.817+0000 7f7b083250c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:14.821 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.821+0000 7f7b083250c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-09T18:09:14.821 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.821+0000 7f7b083250c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:14.822 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.821+0000 7f7b083250c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-09T18:09:14.831 INFO:teuthology.orchestra.run.smithi051.stdout:{ 2020-11-09T18:09:14.831 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "90fe8498-2ca2-45cd-aec4-98a2c19bc03a", 2020-11-09T18:09:14.831 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "default", 2020-11-09T18:09:14.831 INFO:teuthology.orchestra.run.smithi051.stdout: "domain_root": "default.rgw.meta:root", 2020-11-09T18:09:14.831 INFO:teuthology.orchestra.run.smithi051.stdout: "control_pool": "default.rgw.control", 2020-11-09T18:09:14.832 INFO:teuthology.orchestra.run.smithi051.stdout: "gc_pool": "default.rgw.log:gc", 2020-11-09T18:09:14.832 INFO:teuthology.orchestra.run.smithi051.stdout: "lc_pool": "default.rgw.log:lc", 2020-11-09T18:09:14.832 INFO:teuthology.orchestra.run.smithi051.stdout: "log_pool": "default.rgw.log", 2020-11-09T18:09:14.832 INFO:teuthology.orchestra.run.smithi051.stdout: "intent_log_pool": "default.rgw.log:intent", 2020-11-09T18:09:14.832 INFO:teuthology.orchestra.run.smithi051.stdout: "usage_log_pool": "default.rgw.log:usage", 2020-11-09T18:09:14.832 INFO:teuthology.orchestra.run.smithi051.stdout: "roles_pool": "default.rgw.meta:roles", 2020-11-09T18:09:14.832 INFO:teuthology.orchestra.run.smithi051.stdout: "reshard_pool": "default.rgw.log:reshard", 2020-11-09T18:09:14.833 INFO:teuthology.orchestra.run.smithi051.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2020-11-09T18:09:14.833 INFO:teuthology.orchestra.run.smithi051.stdout: "user_email_pool": "default.rgw.meta:users.email", 2020-11-09T18:09:14.833 INFO:teuthology.orchestra.run.smithi051.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2020-11-09T18:09:14.833 INFO:teuthology.orchestra.run.smithi051.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2020-11-09T18:09:14.833 INFO:teuthology.orchestra.run.smithi051.stdout: "otp_pool": "default.rgw.otp", 2020-11-09T18:09:14.833 INFO:teuthology.orchestra.run.smithi051.stdout: "system_key": { 2020-11-09T18:09:14.833 INFO:teuthology.orchestra.run.smithi051.stdout: "access_key": "", 2020-11-09T18:09:14.834 INFO:teuthology.orchestra.run.smithi051.stdout: "secret_key": "" 2020-11-09T18:09:14.834 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:09:14.834 INFO:teuthology.orchestra.run.smithi051.stdout: "placement_pools": [ 2020-11-09T18:09:14.834 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:09:14.834 INFO:teuthology.orchestra.run.smithi051.stdout: "key": "default-placement", 2020-11-09T18:09:14.834 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:09:14.834 INFO:teuthology.orchestra.run.smithi051.stdout: "index_pool": "default.rgw.buckets.index", 2020-11-09T18:09:14.835 INFO:teuthology.orchestra.run.smithi051.stdout: "storage_classes": { 2020-11-09T18:09:14.835 INFO:teuthology.orchestra.run.smithi051.stdout: "STANDARD": { 2020-11-09T18:09:14.835 INFO:teuthology.orchestra.run.smithi051.stdout: "data_pool": "default.rgw.buckets.data", 2020-11-09T18:09:14.835 INFO:teuthology.orchestra.run.smithi051.stdout: "compression_type": "random" 2020-11-09T18:09:14.835 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:09:14.835 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:09:14.835 INFO:teuthology.orchestra.run.smithi051.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2020-11-09T18:09:14.835 INFO:teuthology.orchestra.run.smithi051.stdout: "index_type": 0 2020-11-09T18:09:14.836 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:09:14.836 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:09:14.836 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:09:14.836 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_id": "", 2020-11-09T18:09:14.836 INFO:teuthology.orchestra.run.smithi051.stdout: "notif_pool": "default.rgw.log:notif" 2020-11-09T18:09:14.836 INFO:teuthology.orchestra.run.smithi051.stdout:} 2020-11-09T18:09:14.837 DEBUG:tasks.util.rgw: json result: {'id': '90fe8498-2ca2-45cd-aec4-98a2c19bc03a', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2020-11-09T18:09:14.838 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2020-11-09T18:09:14.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2020-11-09T18:09:14.838 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2020-11-09T18:09:14.918 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:09:14.918 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:09:14.931 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.929+0000 7fe587aae0c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:14.933 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.933+0000 7fe587aae0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:14.934 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.933+0000 7fe587aae0c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:14.934 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.933+0000 7fe587aae0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:14.934 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.933+0000 7fe587aae0c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:14.935 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.933+0000 7fe587aae0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:14.935 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.933+0000 7fe587aae0c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:14.935 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.933+0000 7fe587aae0c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-09T18:09:14.958 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.957+0000 7fe587aae0c0 20 RGWRados::pool_iterate: got zonegroup_info.74d9a742-0ca5-43e1-91f8-76a60a62fa01 2020-11-09T18:09:14.959 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.957+0000 7fe587aae0c0 20 RGWRados::pool_iterate: got zone_info.90fe8498-2ca2-45cd-aec4-98a2c19bc03a 2020-11-09T18:09:14.959 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.957+0000 7fe587aae0c0 20 RGWRados::pool_iterate: got zone_names.default 2020-11-09T18:09:14.959 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.957+0000 7fe587aae0c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2020-11-09T18:09:14.959 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.957+0000 7fe587aae0c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:14.960 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.957+0000 7fe587aae0c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-09T18:09:14.960 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.957+0000 7fe587aae0c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:14.960 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.957+0000 7fe587aae0c0 20 rados_obj.operate() r=0 bl.length=889 2020-11-09T18:09:14.960 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.957+0000 7fe587aae0c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:14.961 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.961+0000 7fe587aae0c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-09T18:09:14.961 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.961+0000 7fe587aae0c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:14.961 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.961+0000 7fe587aae0c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-09T18:09:14.962 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.961+0000 7fe587aae0c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:14.962 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.961+0000 7fe587aae0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:14.962 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.961+0000 7fe587aae0c0 10 cannot find current period zonegroup using local zonegroup 2020-11-09T18:09:14.963 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.961+0000 7fe587aae0c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:14.963 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.961+0000 7fe587aae0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:14.963 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.961+0000 7fe587aae0c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:14.964 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.961+0000 7fe587aae0c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-09T18:09:14.964 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.961+0000 7fe587aae0c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:14.965 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.965+0000 7fe587aae0c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-09T18:09:14.965 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.965+0000 7fe587aae0c0 20 zonegroup default 2020-11-09T18:09:14.965 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.965+0000 7fe587aae0c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:14.965 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.965+0000 7fe587aae0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:14.966 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.965+0000 7fe587aae0c0 10 Cannot find current period zone using local zone 2020-11-09T18:09:14.966 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.965+0000 7fe587aae0c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:14.966 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.965+0000 7fe587aae0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:14.966 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.965+0000 7fe587aae0c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:14.967 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.965+0000 7fe587aae0c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-09T18:09:14.967 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.965+0000 7fe587aae0c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:14.968 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.965+0000 7fe587aae0c0 20 rados_obj.operate() r=0 bl.length=889 2020-11-09T18:09:14.968 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.965+0000 7fe587aae0c0 20 zone default found 2020-11-09T18:09:14.968 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.965+0000 7fe587aae0c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:14.969 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.969+0000 7fe587aae0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:14.969 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.969+0000 7fe587aae0c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:14.970 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.969+0000 7fe587aae0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:14.970 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.969+0000 7fe587aae0c0 20 started sync module instance, tier type = 2020-11-09T18:09:14.970 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:14.969+0000 7fe587aae0c0 20 started zone id=90fe8498-2ca2-45cd-aec4-98a2c19bc03a (name=default) with tier type = 2020-11-09T18:09:15.007 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.005+0000 7fe587aae0c0 20 add_watcher() i=0 2020-11-09T18:09:15.008 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.005+0000 7fe587aae0c0 20 add_watcher() i=1 2020-11-09T18:09:15.008 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.005+0000 7fe587aae0c0 20 add_watcher() i=2 2020-11-09T18:09:15.009 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.005+0000 7fe587aae0c0 20 add_watcher() i=3 2020-11-09T18:09:15.009 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.005+0000 7fe587aae0c0 20 add_watcher() i=4 2020-11-09T18:09:15.009 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.005+0000 7fe587aae0c0 20 add_watcher() i=5 2020-11-09T18:09:15.010 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.005+0000 7fe587aae0c0 20 add_watcher() i=6 2020-11-09T18:09:15.010 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.009+0000 7fe587aae0c0 20 add_watcher() i=7 2020-11-09T18:09:15.011 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.009+0000 7fe587aae0c0 2 all 8 watchers are set, enabling cache 2020-11-09T18:09:15.013 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.013+0000 7fe587aae0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T18:09:15.014 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.013+0000 7fe587aae0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1165 entering: tid=1 2020-11-09T18:09:15.014 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.013+0000 7fe587aae0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1187 max_entries=1 tid=1 2020-11-09T18:09:15.015 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.013+0000 7fe587aae0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1199 missing part, rereading metadata tid= 1 2020-11-09T18:09:15.015 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.013+0000 7fe587aae0c0 20 int rgw::cls::fifo::FIFO::read_meta(uint64_t, optional_yield):934 entering: tid=1 2020-11-09T18:09:15.015 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.013+0000 7fe587aae0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1221 assuming part was not written yet, so end of data: tid=1 2020-11-09T18:09:15.016 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.013+0000 7fe587aae0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T18:09:15.016 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.013+0000 7fe587aae0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1165 entering: tid=1 2020-11-09T18:09:15.016 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.013+0000 7fe587aae0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1187 max_entries=1 tid=1 2020-11-09T18:09:15.017 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.017+0000 7fe587aae0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1199 missing part, rereading metadata tid= 1 2020-11-09T18:09:15.017 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.017+0000 7fe587aae0c0 20 int rgw::cls::fifo::FIFO::read_meta(uint64_t, optional_yield):934 entering: tid=1 2020-11-09T18:09:15.018 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.017+0000 7fe587aae0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1221 assuming part was not written yet, so end of data: tid=1 2020-11-09T18:09:15.018 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.017+0000 7fe587aae0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T18:09:15.019 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.017+0000 7fe587aae0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1165 entering: tid=1 2020-11-09T18:09:15.019 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.017+0000 7fe587aae0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1187 max_entries=1 tid=1 2020-11-09T18:09:15.019 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.017+0000 7fe587aae0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1199 missing part, rereading metadata tid= 1 2020-11-09T18:09:15.020 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.017+0000 7fe587aae0c0 20 int rgw::cls::fifo::FIFO::read_meta(uint64_t, optional_yield):934 entering: tid=1 2020-11-09T18:09:15.020 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.017+0000 7fe587aae0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1221 assuming part was not written yet, so end of data: tid=1 2020-11-09T18:09:15.020 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.017+0000 7fe587aae0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T18:09:15.021 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.021+0000 7fe587aae0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1165 entering: tid=1 2020-11-09T18:09:15.021 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.021+0000 7fe587aae0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1187 max_entries=1 tid=1 2020-11-09T18:09:15.021 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.021+0000 7fe587aae0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1199 missing part, rereading metadata tid= 1 2020-11-09T18:09:15.022 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.021+0000 7fe587aae0c0 20 int rgw::cls::fifo::FIFO::read_meta(uint64_t, optional_yield):934 entering: tid=1 2020-11-09T18:09:15.022 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.021+0000 7fe587aae0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1221 assuming part was not written yet, so end of data: tid=1 2020-11-09T18:09:15.023 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.021+0000 7fe587aae0c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-09T18:09:15.025 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.025+0000 7fe587aae0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T18:09:15.026 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.025+0000 7fe587aae0c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-09T18:09:15.028 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.025+0000 7fe587aae0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T18:09:15.028 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.029+0000 7fe587aae0c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-09T18:09:15.031 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.029+0000 7fe587aae0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T18:09:15.032 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.029+0000 7fe587aae0c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-09T18:09:15.035 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.033+0000 7fe587aae0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T18:09:15.036 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.033+0000 7fe543fef700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-09T18:09:15.037 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.037+0000 7fe587aae0c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-09T18:09:15.037 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.037+0000 7fe587aae0c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-09T18:09:15.138 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.133+0000 7fe4b2ffd700 20 reqs_thread_entry: start 2020-11-09T18:09:15.139 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.133+0000 7fe4b0ff9700 20 reqs_thread_entry: start 2020-11-09T18:09:15.139 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.133+0000 7fe587aae0c0 20 init_complete bucket index max shards: 11 2020-11-09T18:09:15.139 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.133+0000 7fe587aae0c0 10 Started notification manager with: 1 workers 2020-11-09T18:09:15.139 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.133+0000 7fe4a6ffd700 20 INFO: next queues processing will happen at: Mon Nov 9 18:09:45 2020 2020-11-09T18:09:15.139 INFO:teuthology.orchestra.run.smithi051.stderr: 2020-11-09T18:09:15.140 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.133+0000 7fe4a67fc700 20 reqs_thread_entry: start 2020-11-09T18:09:15.162 INFO:teuthology.orchestra.run.smithi051.stdout:[] 2020-11-09T18:09:15.163 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.161+0000 7fe587aae0c0 20 remove_watcher() i=0 2020-11-09T18:09:15.163 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.161+0000 7fe587aae0c0 2 removed watcher, disabling cache 2020-11-09T18:09:15.165 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.165+0000 7fe587aae0c0 20 remove_watcher() i=1 2020-11-09T18:09:15.168 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.165+0000 7fe587aae0c0 20 remove_watcher() i=2 2020-11-09T18:09:15.171 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.169+0000 7fe587aae0c0 20 remove_watcher() i=3 2020-11-09T18:09:15.173 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.173+0000 7fe587aae0c0 20 remove_watcher() i=4 2020-11-09T18:09:15.176 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.173+0000 7fe587aae0c0 20 remove_watcher() i=5 2020-11-09T18:09:15.179 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.177+0000 7fe587aae0c0 20 remove_watcher() i=6 2020-11-09T18:09:15.182 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.181+0000 7fe587aae0c0 20 remove_watcher() i=7 2020-11-09T18:09:15.183 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.181+0000 7fe543fef700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-09T18:09:15.191 DEBUG:tasks.util.rgw: json result: [] 2020-11-09T18:09:15.192 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-09T18:09:15.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-09T18:09:15.192 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-11-09T18:09:15.276 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:09:15.276 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:09:15.289 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.289+0000 7f0a58f140c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-09T18:09:15.290 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.289+0000 7f0a58f140c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-09T18:09:15.297 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.297+0000 7f098e7fc700 20 reqs_thread_entry: start 2020-11-09T18:09:15.297 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.297+0000 7f0a58f140c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:15.299 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.297+0000 7f0a58f140c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-09T18:09:15.299 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.297+0000 7f0a58f140c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:15.302 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.301+0000 7f0a58f140c0 20 rados_obj.operate() r=0 bl.length=889 2020-11-09T18:09:15.303 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.301+0000 7f0a58f140c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:15.303 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.301+0000 7f0a58f140c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:15.303 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.301+0000 7f0a58f140c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:15.304 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.301+0000 7f0a58f140c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-09T18:09:15.304 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.301+0000 7f0a58f140c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:15.304 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:15.301+0000 7f0a58f140c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-09T18:09:15.315 INFO:teuthology.orchestra.run.smithi051.stdout:{ 2020-11-09T18:09:15.315 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "90fe8498-2ca2-45cd-aec4-98a2c19bc03a", 2020-11-09T18:09:15.315 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "default", 2020-11-09T18:09:15.316 INFO:teuthology.orchestra.run.smithi051.stdout: "domain_root": "default.rgw.meta:root", 2020-11-09T18:09:15.316 INFO:teuthology.orchestra.run.smithi051.stdout: "control_pool": "default.rgw.control", 2020-11-09T18:09:15.316 INFO:teuthology.orchestra.run.smithi051.stdout: "gc_pool": "default.rgw.log:gc", 2020-11-09T18:09:15.316 INFO:teuthology.orchestra.run.smithi051.stdout: "lc_pool": "default.rgw.log:lc", 2020-11-09T18:09:15.316 INFO:teuthology.orchestra.run.smithi051.stdout: "log_pool": "default.rgw.log", 2020-11-09T18:09:15.316 INFO:teuthology.orchestra.run.smithi051.stdout: "intent_log_pool": "default.rgw.log:intent", 2020-11-09T18:09:15.316 INFO:teuthology.orchestra.run.smithi051.stdout: "usage_log_pool": "default.rgw.log:usage", 2020-11-09T18:09:15.317 INFO:teuthology.orchestra.run.smithi051.stdout: "roles_pool": "default.rgw.meta:roles", 2020-11-09T18:09:15.317 INFO:teuthology.orchestra.run.smithi051.stdout: "reshard_pool": "default.rgw.log:reshard", 2020-11-09T18:09:15.317 INFO:teuthology.orchestra.run.smithi051.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2020-11-09T18:09:15.317 INFO:teuthology.orchestra.run.smithi051.stdout: "user_email_pool": "default.rgw.meta:users.email", 2020-11-09T18:09:15.317 INFO:teuthology.orchestra.run.smithi051.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2020-11-09T18:09:15.317 INFO:teuthology.orchestra.run.smithi051.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2020-11-09T18:09:15.317 INFO:teuthology.orchestra.run.smithi051.stdout: "otp_pool": "default.rgw.otp", 2020-11-09T18:09:15.317 INFO:teuthology.orchestra.run.smithi051.stdout: "system_key": { 2020-11-09T18:09:15.318 INFO:teuthology.orchestra.run.smithi051.stdout: "access_key": "", 2020-11-09T18:09:15.318 INFO:teuthology.orchestra.run.smithi051.stdout: "secret_key": "" 2020-11-09T18:09:15.318 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:09:15.318 INFO:teuthology.orchestra.run.smithi051.stdout: "placement_pools": [ 2020-11-09T18:09:15.318 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:09:15.318 INFO:teuthology.orchestra.run.smithi051.stdout: "key": "default-placement", 2020-11-09T18:09:15.319 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:09:15.319 INFO:teuthology.orchestra.run.smithi051.stdout: "index_pool": "default.rgw.buckets.index", 2020-11-09T18:09:15.319 INFO:teuthology.orchestra.run.smithi051.stdout: "storage_classes": { 2020-11-09T18:09:15.319 INFO:teuthology.orchestra.run.smithi051.stdout: "STANDARD": { 2020-11-09T18:09:15.319 INFO:teuthology.orchestra.run.smithi051.stdout: "data_pool": "default.rgw.buckets.data", 2020-11-09T18:09:15.319 INFO:teuthology.orchestra.run.smithi051.stdout: "compression_type": "random" 2020-11-09T18:09:15.319 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:09:15.320 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:09:15.320 INFO:teuthology.orchestra.run.smithi051.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2020-11-09T18:09:15.320 INFO:teuthology.orchestra.run.smithi051.stdout: "index_type": 0 2020-11-09T18:09:15.320 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:09:15.320 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:09:15.320 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:09:15.320 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_id": "", 2020-11-09T18:09:15.320 INFO:teuthology.orchestra.run.smithi051.stdout: "notif_pool": "default.rgw.log:notif" 2020-11-09T18:09:15.321 INFO:teuthology.orchestra.run.smithi051.stdout:} 2020-11-09T18:09:15.321 DEBUG:tasks.util.rgw: json result: {'id': '90fe8498-2ca2-45cd-aec4-98a2c19bc03a', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2020-11-09T18:09:15.322 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2020-11-09T18:09:15.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2020-11-09T18:09:15.324 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2020-11-09T18:09:15.363 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:09:15.364 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:09:15.376 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:15.371+0000 7f262a4a30c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:16.488 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:16.483+0000 7f262a4a30c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:16.488 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:16.483+0000 7f262a4a30c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:16.488 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:16.483+0000 7f262a4a30c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:16.489 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:16.483+0000 7f262a4a30c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:16.489 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:16.487+0000 7f262a4a30c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:16.489 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:16.487+0000 7f262a4a30c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:16.489 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:16.487+0000 7f262a4a30c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:16.514 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:16.511+0000 7f262a4a30c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:16.515 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:16.511+0000 7f262a4a30c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:16.515 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:16.511+0000 7f262a4a30c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:16.516 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:16.511+0000 7f262a4a30c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:16.516 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:16.511+0000 7f262a4a30c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:16.516 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:16.511+0000 7f262a4a30c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:16.516 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:16.511+0000 7f262a4a30c0 10 cannot find current period zonegroup using local zonegroup 2020-11-09T18:09:16.517 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:16.511+0000 7f262a4a30c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:16.517 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:16.511+0000 7f262a4a30c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:16.517 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:16.511+0000 7f262a4a30c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:16.517 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:16.511+0000 7f262a4a30c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:16.517 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:16.511+0000 7f262a4a30c0 10 Creating default zonegroup 2020-11-09T18:09:16.517 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:16.511+0000 7f262a4a30c0 20 get_system_obj_state: rctx=0x7ffc13476b90 obj=:.pools.avail state=0x5646447cf2e0 s->prefetch_data=0 2020-11-09T18:09:16.517 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:16.511+0000 7f262a4a30c0 10 couldn't find old data placement pools config, setting up new ones for the zone 2020-11-09T18:09:16.531 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:16.527+0000 7f262a4a30c0 10 WARNING: store->list_zones() returned r=-2 2020-11-09T18:09:16.531 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:16.527+0000 7f262a4a30c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:16.532 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:16.527+0000 7f262a4a30c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:16.536 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:16.535+0000 7f262a4a30c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:16.537 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:16.535+0000 7f262a4a30c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:16.537 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:16.535+0000 7f262a4a30c0 10 could not read realm id: (2) No such file or directory 2020-11-09T18:09:16.537 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:16.535+0000 7f262a4a30c0 10 WARNING: failed to set zone as default, r=-22 2020-11-09T18:09:16.537 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:16.535+0000 7f262a4a30c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:16.538 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:16.535+0000 7f262a4a30c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:16.543 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:16.539+0000 7f262a4a30c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:16.543 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:16.539+0000 7f262a4a30c0 20 rados_obj.operate() r=0 bl.length=873 2020-11-09T18:09:16.544 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:16.539+0000 7f262a4a30c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:16.544 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:16.539+0000 7f262a4a30c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-09T18:09:16.544 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:16.539+0000 7f262a4a30c0 20 zonegroup default 2020-11-09T18:09:16.545 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:16.539+0000 7f262a4a30c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:16.545 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:16.543+0000 7f262a4a30c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:16.545 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:16.543+0000 7f262a4a30c0 10 Cannot find current period zone using local zone 2020-11-09T18:09:16.546 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:16.543+0000 7f262a4a30c0 10 Using default name default 2020-11-09T18:09:16.546 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:16.543+0000 7f262a4a30c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:16.546 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:16.543+0000 7f262a4a30c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-09T18:09:16.547 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:16.543+0000 7f262a4a30c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:16.547 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:16.543+0000 7f262a4a30c0 20 rados_obj.operate() r=0 bl.length=873 2020-11-09T18:09:16.547 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:16.543+0000 7f262a4a30c0 20 zone default found 2020-11-09T18:09:16.547 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:16.543+0000 7f262a4a30c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:18.506 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:18.503+0000 7f262a4a30c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:18.506 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:18.503+0000 7f262a4a30c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:18.506 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:18.503+0000 7f262a4a30c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:18.507 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:18.503+0000 7f262a4a30c0 20 started sync module instance, tier type = 2020-11-09T18:09:18.507 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:18.503+0000 7f262a4a30c0 20 started zone id=4d7e4c32-4d6a-4e02-92c3-8484e0b4b2ab (name=default) with tier type = 2020-11-09T18:09:20.572 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:20.567+0000 7f262a4a30c0 20 add_watcher() i=0 2020-11-09T18:09:20.573 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:20.567+0000 7f262a4a30c0 20 add_watcher() i=1 2020-11-09T18:09:20.573 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:20.567+0000 7f262a4a30c0 20 add_watcher() i=2 2020-11-09T18:09:20.573 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:20.567+0000 7f262a4a30c0 20 add_watcher() i=3 2020-11-09T18:09:20.573 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:20.567+0000 7f262a4a30c0 20 add_watcher() i=4 2020-11-09T18:09:20.573 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:20.567+0000 7f262a4a30c0 20 add_watcher() i=5 2020-11-09T18:09:20.573 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:20.567+0000 7f262a4a30c0 20 add_watcher() i=6 2020-11-09T18:09:20.574 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:20.571+0000 7f262a4a30c0 20 add_watcher() i=7 2020-11-09T18:09:20.575 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:20.571+0000 7f262a4a30c0 2 all 8 watchers are set, enabling cache 2020-11-09T18:09:20.577 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:20.575+0000 7f262a4a30c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T18:09:20.578 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:20.575+0000 7f262a4a30c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T18:09:20.578 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:20.575+0000 7f262a4a30c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T18:09:20.579 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:20.575+0000 7f262a4a30c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T18:09:20.579 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:20.575+0000 7f262a4a30c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-09T18:09:20.585 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:20.583+0000 7f262a4a30c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T18:09:20.589 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:20.587+0000 7f262a4a30c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-09T18:09:20.593 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:20.591+0000 7f262a4a30c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T18:09:20.599 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:20.591+0000 7f262a4a30c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-09T18:09:20.600 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:20.595+0000 7f262a4a30c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T18:09:20.603 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:20.599+0000 7f262a4a30c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-09T18:09:20.607 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:20.603+0000 7f262a4a30c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T18:09:20.611 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:20.607+0000 7f25e6fed700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-09T18:09:20.611 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:20.607+0000 7f262a4a30c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-09T18:09:20.611 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:20.607+0000 7f262a4a30c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-09T18:09:24.541 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.531+0000 7f262a4a30c0 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2020-11-09T18:09:24.648 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.643+0000 7f2555ffb700 20 reqs_thread_entry: start 2020-11-09T18:09:24.648 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.643+0000 7f262a4a30c0 20 init_complete bucket index max shards: 11 2020-11-09T18:09:24.649 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.643+0000 7f253b7fe700 20 reqs_thread_entry: start 2020-11-09T18:09:24.649 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.643+0000 7f262a4a30c0 10 Started notification manager with: 1 workers 2020-11-09T18:09:24.649 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.643+0000 7f2539ffb700 20 INFO: next queues processing will happen at: Mon Nov 9 18:09:54 2020 2020-11-09T18:09:24.649 INFO:teuthology.orchestra.run.smithi078.stderr: 2020-11-09T18:09:24.649 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.643+0000 7f25397fa700 20 reqs_thread_entry: start 2020-11-09T18:09:24.690 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2020-11-09T18:09:24.691 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.675+0000 7f262a4a30c0 20 remove_watcher() i=0 2020-11-09T18:09:24.691 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.675+0000 7f262a4a30c0 2 removed watcher, disabling cache 2020-11-09T18:09:24.691 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.679+0000 7f262a4a30c0 20 remove_watcher() i=1 2020-11-09T18:09:24.691 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.679+0000 7f262a4a30c0 20 remove_watcher() i=2 2020-11-09T18:09:24.692 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.687+0000 7f262a4a30c0 20 remove_watcher() i=3 2020-11-09T18:09:24.692 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.687+0000 7f262a4a30c0 20 remove_watcher() i=4 2020-11-09T18:09:24.695 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.691+0000 7f262a4a30c0 20 remove_watcher() i=5 2020-11-09T18:09:24.698 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.695+0000 7f262a4a30c0 20 remove_watcher() i=6 2020-11-09T18:09:24.701 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.699+0000 7f262a4a30c0 20 remove_watcher() i=7 2020-11-09T18:09:24.702 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.699+0000 7f25e6fed700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-09T18:09:24.720 DEBUG:tasks.util.rgw: json result: [] 2020-11-09T18:09:24.720 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-09T18:09:24.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-09T18:09:24.721 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-11-09T18:09:24.801 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:09:24.802 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:09:24.814 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.807+0000 7f9267d710c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-09T18:09:24.814 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.807+0000 7f9267d710c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-09T18:09:24.820 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.815+0000 7f91a4ff9700 20 reqs_thread_entry: start 2020-11-09T18:09:24.820 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.815+0000 7f9267d710c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:24.820 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.815+0000 7f9267d710c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-09T18:09:24.821 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.815+0000 7f9267d710c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:24.821 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.819+0000 7f9267d710c0 20 rados_obj.operate() r=0 bl.length=873 2020-11-09T18:09:24.822 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.819+0000 7f9267d710c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:24.822 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.819+0000 7f9267d710c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:24.822 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.819+0000 7f9267d710c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:24.823 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.819+0000 7f9267d710c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-09T18:09:24.823 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.819+0000 7f9267d710c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:24.824 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.819+0000 7f9267d710c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-09T18:09:24.831 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:09:24.832 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "4d7e4c32-4d6a-4e02-92c3-8484e0b4b2ab", 2020-11-09T18:09:24.832 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "default", 2020-11-09T18:09:24.832 INFO:teuthology.orchestra.run.smithi078.stdout: "domain_root": "default.rgw.meta:root", 2020-11-09T18:09:24.832 INFO:teuthology.orchestra.run.smithi078.stdout: "control_pool": "default.rgw.control", 2020-11-09T18:09:24.832 INFO:teuthology.orchestra.run.smithi078.stdout: "gc_pool": "default.rgw.log:gc", 2020-11-09T18:09:24.833 INFO:teuthology.orchestra.run.smithi078.stdout: "lc_pool": "default.rgw.log:lc", 2020-11-09T18:09:24.833 INFO:teuthology.orchestra.run.smithi078.stdout: "log_pool": "default.rgw.log", 2020-11-09T18:09:24.833 INFO:teuthology.orchestra.run.smithi078.stdout: "intent_log_pool": "default.rgw.log:intent", 2020-11-09T18:09:24.833 INFO:teuthology.orchestra.run.smithi078.stdout: "usage_log_pool": "default.rgw.log:usage", 2020-11-09T18:09:24.833 INFO:teuthology.orchestra.run.smithi078.stdout: "roles_pool": "default.rgw.meta:roles", 2020-11-09T18:09:24.834 INFO:teuthology.orchestra.run.smithi078.stdout: "reshard_pool": "default.rgw.log:reshard", 2020-11-09T18:09:24.834 INFO:teuthology.orchestra.run.smithi078.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2020-11-09T18:09:24.834 INFO:teuthology.orchestra.run.smithi078.stdout: "user_email_pool": "default.rgw.meta:users.email", 2020-11-09T18:09:24.834 INFO:teuthology.orchestra.run.smithi078.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2020-11-09T18:09:24.834 INFO:teuthology.orchestra.run.smithi078.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2020-11-09T18:09:24.834 INFO:teuthology.orchestra.run.smithi078.stdout: "otp_pool": "default.rgw.otp", 2020-11-09T18:09:24.834 INFO:teuthology.orchestra.run.smithi078.stdout: "system_key": { 2020-11-09T18:09:24.835 INFO:teuthology.orchestra.run.smithi078.stdout: "access_key": "", 2020-11-09T18:09:24.835 INFO:teuthology.orchestra.run.smithi078.stdout: "secret_key": "" 2020-11-09T18:09:24.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:09:24.835 INFO:teuthology.orchestra.run.smithi078.stdout: "placement_pools": [ 2020-11-09T18:09:24.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:09:24.835 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "default-placement", 2020-11-09T18:09:24.835 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:09:24.836 INFO:teuthology.orchestra.run.smithi078.stdout: "index_pool": "default.rgw.buckets.index", 2020-11-09T18:09:24.836 INFO:teuthology.orchestra.run.smithi078.stdout: "storage_classes": { 2020-11-09T18:09:24.836 INFO:teuthology.orchestra.run.smithi078.stdout: "STANDARD": { 2020-11-09T18:09:24.836 INFO:teuthology.orchestra.run.smithi078.stdout: "data_pool": "default.rgw.buckets.data", 2020-11-09T18:09:24.836 INFO:teuthology.orchestra.run.smithi078.stdout: "compression_type": "random" 2020-11-09T18:09:24.836 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:09:24.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:09:24.837 INFO:teuthology.orchestra.run.smithi078.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2020-11-09T18:09:24.837 INFO:teuthology.orchestra.run.smithi078.stdout: "index_type": 0 2020-11-09T18:09:24.837 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:09:24.837 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:09:24.837 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:09:24.837 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_id": "", 2020-11-09T18:09:24.837 INFO:teuthology.orchestra.run.smithi078.stdout: "notif_pool": "default.rgw.log:notif" 2020-11-09T18:09:24.838 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:09:24.838 DEBUG:tasks.util.rgw: json result: {'id': '4d7e4c32-4d6a-4e02-92c3-8484e0b4b2ab', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2020-11-09T18:09:24.840 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2020-11-09T18:09:24.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2020-11-09T18:09:24.840 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2020-11-09T18:09:24.914 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:09:24.915 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:09:24.924 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.919+0000 7f1ef3d310c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:24.928 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.923+0000 7f1ef3d310c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:24.928 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.923+0000 7f1ef3d310c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:24.929 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.927+0000 7f1ef3d310c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:24.929 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.927+0000 7f1ef3d310c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:24.930 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.927+0000 7f1ef3d310c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:24.930 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.927+0000 7f1ef3d310c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:24.930 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.927+0000 7f1ef3d310c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-09T18:09:24.951 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.947+0000 7f1ef3d310c0 20 RGWRados::pool_iterate: got zonegroup_info.d83c1cdb-ed04-4e91-b654-32d80fac4c7b 2020-11-09T18:09:24.952 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.947+0000 7f1ef3d310c0 20 RGWRados::pool_iterate: got zone_info.4d7e4c32-4d6a-4e02-92c3-8484e0b4b2ab 2020-11-09T18:09:24.952 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.947+0000 7f1ef3d310c0 20 RGWRados::pool_iterate: got zone_names.default 2020-11-09T18:09:24.952 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.947+0000 7f1ef3d310c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2020-11-09T18:09:24.952 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.947+0000 7f1ef3d310c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:24.953 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.947+0000 7f1ef3d310c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-09T18:09:24.953 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.947+0000 7f1ef3d310c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:24.953 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.951+0000 7f1ef3d310c0 20 rados_obj.operate() r=0 bl.length=889 2020-11-09T18:09:24.953 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.951+0000 7f1ef3d310c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:24.954 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.951+0000 7f1ef3d310c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-09T18:09:24.954 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.951+0000 7f1ef3d310c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:24.954 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.951+0000 7f1ef3d310c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-09T18:09:24.955 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.951+0000 7f1ef3d310c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:24.955 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.951+0000 7f1ef3d310c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:24.955 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.951+0000 7f1ef3d310c0 10 cannot find current period zonegroup using local zonegroup 2020-11-09T18:09:24.955 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.951+0000 7f1ef3d310c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:24.956 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.951+0000 7f1ef3d310c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:24.956 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.951+0000 7f1ef3d310c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:24.956 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.951+0000 7f1ef3d310c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-09T18:09:24.956 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.951+0000 7f1ef3d310c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:24.957 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.955+0000 7f1ef3d310c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-09T18:09:24.957 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.955+0000 7f1ef3d310c0 20 zonegroup default 2020-11-09T18:09:24.957 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.955+0000 7f1ef3d310c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:24.958 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.955+0000 7f1ef3d310c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:24.958 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.955+0000 7f1ef3d310c0 10 Cannot find current period zone using local zone 2020-11-09T18:09:24.958 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.955+0000 7f1ef3d310c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:24.958 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.955+0000 7f1ef3d310c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:24.959 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.955+0000 7f1ef3d310c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:24.959 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.955+0000 7f1ef3d310c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-09T18:09:24.959 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.955+0000 7f1ef3d310c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:24.959 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.955+0000 7f1ef3d310c0 20 rados_obj.operate() r=0 bl.length=889 2020-11-09T18:09:24.960 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.955+0000 7f1ef3d310c0 20 zone default found 2020-11-09T18:09:24.960 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.955+0000 7f1ef3d310c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:24.960 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.955+0000 7f1ef3d310c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:24.960 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.955+0000 7f1ef3d310c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:24.961 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.959+0000 7f1ef3d310c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:24.961 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.959+0000 7f1ef3d310c0 20 started sync module instance, tier type = 2020-11-09T18:09:24.961 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.959+0000 7f1ef3d310c0 20 started zone id=4d7e4c32-4d6a-4e02-92c3-8484e0b4b2ab (name=default) with tier type = 2020-11-09T18:09:24.993 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.991+0000 7f1ef3d310c0 20 add_watcher() i=0 2020-11-09T18:09:24.994 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.991+0000 7f1ef3d310c0 20 add_watcher() i=1 2020-11-09T18:09:24.994 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.991+0000 7f1ef3d310c0 20 add_watcher() i=2 2020-11-09T18:09:24.994 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.991+0000 7f1ef3d310c0 20 add_watcher() i=3 2020-11-09T18:09:24.994 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.991+0000 7f1ef3d310c0 20 add_watcher() i=4 2020-11-09T18:09:24.994 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.991+0000 7f1ef3d310c0 20 add_watcher() i=5 2020-11-09T18:09:24.994 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.991+0000 7f1ef3d310c0 20 add_watcher() i=6 2020-11-09T18:09:24.995 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.991+0000 7f1ef3d310c0 20 add_watcher() i=7 2020-11-09T18:09:24.995 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.991+0000 7f1ef3d310c0 2 all 8 watchers are set, enabling cache 2020-11-09T18:09:24.998 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.995+0000 7f1ef3d310c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T18:09:24.998 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.995+0000 7f1ef3d310c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1165 entering: tid=1 2020-11-09T18:09:24.999 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.995+0000 7f1ef3d310c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1187 max_entries=1 tid=1 2020-11-09T18:09:24.999 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.995+0000 7f1ef3d310c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1199 missing part, rereading metadata tid= 1 2020-11-09T18:09:24.999 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.995+0000 7f1ef3d310c0 20 int rgw::cls::fifo::FIFO::read_meta(uint64_t, optional_yield):934 entering: tid=1 2020-11-09T18:09:25.000 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.995+0000 7f1ef3d310c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1221 assuming part was not written yet, so end of data: tid=1 2020-11-09T18:09:25.000 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.995+0000 7f1ef3d310c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T18:09:25.000 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.999+0000 7f1ef3d310c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1165 entering: tid=1 2020-11-09T18:09:25.001 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.999+0000 7f1ef3d310c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1187 max_entries=1 tid=1 2020-11-09T18:09:25.001 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.999+0000 7f1ef3d310c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1199 missing part, rereading metadata tid= 1 2020-11-09T18:09:25.001 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.999+0000 7f1ef3d310c0 20 int rgw::cls::fifo::FIFO::read_meta(uint64_t, optional_yield):934 entering: tid=1 2020-11-09T18:09:25.002 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.999+0000 7f1ef3d310c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1221 assuming part was not written yet, so end of data: tid=1 2020-11-09T18:09:25.002 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.999+0000 7f1ef3d310c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T18:09:25.003 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.999+0000 7f1ef3d310c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1165 entering: tid=1 2020-11-09T18:09:25.003 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.999+0000 7f1ef3d310c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1187 max_entries=1 tid=1 2020-11-09T18:09:25.003 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.999+0000 7f1ef3d310c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1199 missing part, rereading metadata tid= 1 2020-11-09T18:09:25.003 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.999+0000 7f1ef3d310c0 20 int rgw::cls::fifo::FIFO::read_meta(uint64_t, optional_yield):934 entering: tid=1 2020-11-09T18:09:25.004 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.999+0000 7f1ef3d310c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1221 assuming part was not written yet, so end of data: tid=1 2020-11-09T18:09:25.004 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:24.999+0000 7f1ef3d310c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T18:09:25.005 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:25.003+0000 7f1ef3d310c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1165 entering: tid=1 2020-11-09T18:09:25.005 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:25.003+0000 7f1ef3d310c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1187 max_entries=1 tid=1 2020-11-09T18:09:25.005 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:25.003+0000 7f1ef3d310c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1199 missing part, rereading metadata tid= 1 2020-11-09T18:09:25.006 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:25.003+0000 7f1ef3d310c0 20 int rgw::cls::fifo::FIFO::read_meta(uint64_t, optional_yield):934 entering: tid=1 2020-11-09T18:09:25.010 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:25.003+0000 7f1ef3d310c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1221 assuming part was not written yet, so end of data: tid=1 2020-11-09T18:09:25.011 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:25.003+0000 7f1ef3d310c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-09T18:09:25.011 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:25.007+0000 7f1ef3d310c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T18:09:25.011 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:25.007+0000 7f1ef3d310c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-09T18:09:25.014 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:25.011+0000 7f1ef3d310c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T18:09:25.015 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:25.011+0000 7f1ef3d310c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-09T18:09:25.017 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:25.015+0000 7f1ef3d310c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T18:09:25.018 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:25.015+0000 7f1ef3d310c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-09T18:09:25.021 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:25.019+0000 7f1ef3d310c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T18:09:25.022 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:25.019+0000 7f1eb17ea700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-09T18:09:25.022 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:25.019+0000 7f1ef3d310c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-09T18:09:25.022 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:25.019+0000 7f1ef3d310c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-09T18:09:25.123 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:25.119+0000 7f1e1dffb700 20 reqs_thread_entry: start 2020-11-09T18:09:25.123 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:25.119+0000 7f1e1ffff700 20 reqs_thread_entry: start 2020-11-09T18:09:25.123 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:25.119+0000 7f1ef3d310c0 20 init_complete bucket index max shards: 11 2020-11-09T18:09:25.123 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:25.119+0000 7f1ef3d310c0 10 Started notification manager with: 1 workers 2020-11-09T18:09:25.124 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:25.119+0000 7f1e0bfff700 20 INFO: next queues processing will happen at: Mon Nov 9 18:09:55 2020 2020-11-09T18:09:25.124 INFO:teuthology.orchestra.run.smithi078.stderr: 2020-11-09T18:09:25.124 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:25.119+0000 7f1e0b7fe700 20 reqs_thread_entry: start 2020-11-09T18:09:25.154 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2020-11-09T18:09:25.155 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:25.151+0000 7f1ef3d310c0 20 remove_watcher() i=0 2020-11-09T18:09:25.155 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:25.151+0000 7f1ef3d310c0 2 removed watcher, disabling cache 2020-11-09T18:09:25.157 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:25.155+0000 7f1ef3d310c0 20 remove_watcher() i=1 2020-11-09T18:09:25.160 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:25.155+0000 7f1ef3d310c0 20 remove_watcher() i=2 2020-11-09T18:09:25.163 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:25.159+0000 7f1ef3d310c0 20 remove_watcher() i=3 2020-11-09T18:09:25.166 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:25.163+0000 7f1ef3d310c0 20 remove_watcher() i=4 2020-11-09T18:09:25.168 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:25.163+0000 7f1ef3d310c0 20 remove_watcher() i=5 2020-11-09T18:09:25.171 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:25.167+0000 7f1ef3d310c0 20 remove_watcher() i=6 2020-11-09T18:09:25.175 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:25.171+0000 7f1ef3d310c0 20 remove_watcher() i=7 2020-11-09T18:09:25.176 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:25.175+0000 7f1eb17ea700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-09T18:09:25.183 DEBUG:tasks.util.rgw: json result: [] 2020-11-09T18:09:25.183 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-09T18:09:25.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-09T18:09:25.184 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-11-09T18:09:25.266 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:09:25.266 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:09:25.276 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:25.271+0000 7f83b6efe0c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-09T18:09:25.277 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:25.271+0000 7f83b6efe0c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-09T18:09:25.283 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:25.279+0000 7f82e7fff700 20 reqs_thread_entry: start 2020-11-09T18:09:25.283 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:25.279+0000 7f83b6efe0c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:25.285 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:25.283+0000 7f83b6efe0c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-09T18:09:25.285 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:25.283+0000 7f83b6efe0c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:25.286 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:25.283+0000 7f83b6efe0c0 20 rados_obj.operate() r=0 bl.length=889 2020-11-09T18:09:25.286 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:25.283+0000 7f83b6efe0c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:25.287 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:25.283+0000 7f83b6efe0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T18:09:25.287 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:25.283+0000 7f83b6efe0c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:25.287 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:25.283+0000 7f83b6efe0c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-09T18:09:25.288 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:25.283+0000 7f83b6efe0c0 20 rados->read ofs=0 len=0 2020-11-09T18:09:25.288 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:25.283+0000 7f83b6efe0c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-09T18:09:25.296 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:09:25.297 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "4d7e4c32-4d6a-4e02-92c3-8484e0b4b2ab", 2020-11-09T18:09:25.297 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "default", 2020-11-09T18:09:25.297 INFO:teuthology.orchestra.run.smithi078.stdout: "domain_root": "default.rgw.meta:root", 2020-11-09T18:09:25.297 INFO:teuthology.orchestra.run.smithi078.stdout: "control_pool": "default.rgw.control", 2020-11-09T18:09:25.297 INFO:teuthology.orchestra.run.smithi078.stdout: "gc_pool": "default.rgw.log:gc", 2020-11-09T18:09:25.297 INFO:teuthology.orchestra.run.smithi078.stdout: "lc_pool": "default.rgw.log:lc", 2020-11-09T18:09:25.298 INFO:teuthology.orchestra.run.smithi078.stdout: "log_pool": "default.rgw.log", 2020-11-09T18:09:25.298 INFO:teuthology.orchestra.run.smithi078.stdout: "intent_log_pool": "default.rgw.log:intent", 2020-11-09T18:09:25.298 INFO:teuthology.orchestra.run.smithi078.stdout: "usage_log_pool": "default.rgw.log:usage", 2020-11-09T18:09:25.298 INFO:teuthology.orchestra.run.smithi078.stdout: "roles_pool": "default.rgw.meta:roles", 2020-11-09T18:09:25.298 INFO:teuthology.orchestra.run.smithi078.stdout: "reshard_pool": "default.rgw.log:reshard", 2020-11-09T18:09:25.298 INFO:teuthology.orchestra.run.smithi078.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2020-11-09T18:09:25.298 INFO:teuthology.orchestra.run.smithi078.stdout: "user_email_pool": "default.rgw.meta:users.email", 2020-11-09T18:09:25.299 INFO:teuthology.orchestra.run.smithi078.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2020-11-09T18:09:25.299 INFO:teuthology.orchestra.run.smithi078.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2020-11-09T18:09:25.299 INFO:teuthology.orchestra.run.smithi078.stdout: "otp_pool": "default.rgw.otp", 2020-11-09T18:09:25.299 INFO:teuthology.orchestra.run.smithi078.stdout: "system_key": { 2020-11-09T18:09:25.299 INFO:teuthology.orchestra.run.smithi078.stdout: "access_key": "", 2020-11-09T18:09:25.299 INFO:teuthology.orchestra.run.smithi078.stdout: "secret_key": "" 2020-11-09T18:09:25.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:09:25.300 INFO:teuthology.orchestra.run.smithi078.stdout: "placement_pools": [ 2020-11-09T18:09:25.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:09:25.300 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "default-placement", 2020-11-09T18:09:25.300 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:09:25.300 INFO:teuthology.orchestra.run.smithi078.stdout: "index_pool": "default.rgw.buckets.index", 2020-11-09T18:09:25.300 INFO:teuthology.orchestra.run.smithi078.stdout: "storage_classes": { 2020-11-09T18:09:25.300 INFO:teuthology.orchestra.run.smithi078.stdout: "STANDARD": { 2020-11-09T18:09:25.301 INFO:teuthology.orchestra.run.smithi078.stdout: "data_pool": "default.rgw.buckets.data", 2020-11-09T18:09:25.301 INFO:teuthology.orchestra.run.smithi078.stdout: "compression_type": "random" 2020-11-09T18:09:25.301 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:09:25.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:09:25.301 INFO:teuthology.orchestra.run.smithi078.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2020-11-09T18:09:25.301 INFO:teuthology.orchestra.run.smithi078.stdout: "index_type": 0 2020-11-09T18:09:25.301 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:09:25.301 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:09:25.302 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:09:25.302 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_id": "", 2020-11-09T18:09:25.302 INFO:teuthology.orchestra.run.smithi078.stdout: "notif_pool": "default.rgw.log:notif" 2020-11-09T18:09:25.302 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:09:25.303 DEBUG:tasks.util.rgw: json result: {'id': '4d7e4c32-4d6a-4e02-92c3-8484e0b4b2ab', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2020-11-09T18:09:25.303 INFO:tasks.rgw:Starting rgw... 2020-11-09T18:09:25.304 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2020-11-09T18:09:25.304 INFO:tasks.rgw:Using civetweb as radosgw frontend 2020-11-09T18:09:25.304 INFO:tasks.rgw.c1.client.0:Restarting daemon 2020-11-09T18:09:25.305 INFO:teuthology.orchestra.run.smithi051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2020-11-09T18:09:25.307 INFO:tasks.rgw.c1.client.0:Started 2020-11-09T18:09:25.307 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2020-11-09T18:09:25.308 INFO:tasks.rgw:Using civetweb as radosgw frontend 2020-11-09T18:09:25.308 INFO:tasks.rgw.c1.client.1:Restarting daemon 2020-11-09T18:09:25.308 INFO:teuthology.orchestra.run.smithi051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2020-11-09T18:09:25.310 INFO:tasks.rgw.c1.client.1:Started 2020-11-09T18:09:25.311 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2020-11-09T18:09:25.311 INFO:tasks.rgw:Using civetweb as radosgw frontend 2020-11-09T18:09:25.311 INFO:tasks.rgw.c2.client.0:Restarting daemon 2020-11-09T18:09:25.311 INFO:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2020-11-09T18:09:25.350 INFO:tasks.rgw.c2.client.0:Started 2020-11-09T18:09:25.351 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2020-11-09T18:09:25.351 INFO:tasks.rgw:Using civetweb as radosgw frontend 2020-11-09T18:09:25.351 INFO:tasks.rgw.c2.client.1:Restarting daemon 2020-11-09T18:09:25.352 INFO:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2020-11-09T18:09:25.354 INFO:tasks.rgw.c2.client.1:Started 2020-11-09T18:09:25.355 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi051.front.sepia.ceph.com:8000/ 2020-11-09T18:09:25.355 INFO:teuthology.orchestra.run.smithi051:> curl http://smithi051.front.sepia.ceph.com:8000/ 2020-11-09T18:09:25.379 INFO:teuthology.orchestra.run.smithi051.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-09T18:09:25.379 INFO:teuthology.orchestra.run.smithi051.stderr: Dload Upload Total Spent Left Speed 2020-11-09T18:09:25.383 INFO:teuthology.orchestra.run.smithi051.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi051.front.sepia.ceph.com port 8000: Connection refused 2020-11-09T18:09:25.384 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-11-09T18:09:26.385 INFO:teuthology.orchestra.run.smithi051:> curl http://smithi051.front.sepia.ceph.com:8000/ 2020-11-09T18:09:26.400 INFO:teuthology.orchestra.run.smithi051.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-09T18:09:26.400 INFO:teuthology.orchestra.run.smithi051.stderr: Dload Upload Total Spent Left Speed 2020-11-09T18:09:26.405 INFO:teuthology.orchestra.run.smithi051.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 35666 0 --:--:-- --:--:-- --:--:-- 35666 2020-11-09T18:09:26.407 INFO:teuthology.orchestra.run.smithi051.stdout:anonymous 2020-11-09T18:09:26.407 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi051.front.sepia.ceph.com:8001/ 2020-11-09T18:09:26.408 INFO:teuthology.orchestra.run.smithi051:> curl http://smithi051.front.sepia.ceph.com:8001/ 2020-11-09T18:09:26.460 INFO:teuthology.orchestra.run.smithi051.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-09T18:09:26.460 INFO:teuthology.orchestra.run.smithi051.stderr: Dload Upload Total Spent Left Speed 2020-11-09T18:09:26.466 INFO:teuthology.orchestra.run.smithi051.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 35666 0 --:--:-- --:--:-- --:--:-- 35666 2020-11-09T18:09:26.467 INFO:teuthology.orchestra.run.smithi051.stdout:anonymous 2020-11-09T18:09:26.467 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi078.front.sepia.ceph.com:8000/ 2020-11-09T18:09:26.467 INFO:teuthology.orchestra.run.smithi078:> curl http://smithi078.front.sepia.ceph.com:8000/ 2020-11-09T18:09:26.493 INFO:teuthology.orchestra.run.smithi078.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-09T18:09:26.493 INFO:teuthology.orchestra.run.smithi078.stderr: Dload Upload Total Spent Left Speed 2020-11-09T18:09:26.499 INFO:teuthology.orchestra.run.smithi078.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 35666 0 --:--:-- --:--:-- --:--:-- 35666 2020-11-09T18:09:26.499 INFO:teuthology.orchestra.run.smithi078.stdout:anonymous 2020-11-09T18:09:26.500 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi078.front.sepia.ceph.com:8001/ 2020-11-09T18:09:26.500 INFO:teuthology.orchestra.run.smithi078:> curl http://smithi078.front.sepia.ceph.com:8001/ 2020-11-09T18:09:26.557 INFO:teuthology.orchestra.run.smithi078.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-09T18:09:26.558 INFO:teuthology.orchestra.run.smithi078.stderr: Dload Upload Total Spent Left Speed 2020-11-09T18:09:26.564 INFO:teuthology.orchestra.run.smithi078.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 35666 0 --:--:-- --:--:-- --:--:-- 35666 2020-11-09T18:09:26.564 INFO:teuthology.orchestra.run.smithi078.stdout:anonymous 2020-11-09T18:09:26.564 INFO:teuthology.run_tasks:Running task rgw-multisite... 2020-11-09T18:09:26.753 INFO:tasks.rgw_multisite:creating realm.. 2020-11-09T18:09:26.754 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:09:26.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:09:26.755 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-09T18:09:26.792 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:09:26.792 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:09:26.843 INFO:teuthology.orchestra.run.smithi051.stdout:{ 2020-11-09T18:09:26.844 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "0de9c5fb-e243-4abd-b414-59ffbc2cb575", 2020-11-09T18:09:26.844 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "test-realm", 2020-11-09T18:09:26.844 INFO:teuthology.orchestra.run.smithi051.stdout: "current_period": "7de5cbf1-978a-419f-aebf-c2a9e564b43c", 2020-11-09T18:09:26.844 INFO:teuthology.orchestra.run.smithi051.stdout: "epoch": 1 2020-11-09T18:09:26.844 INFO:teuthology.orchestra.run.smithi051.stdout:} 2020-11-09T18:09:26.846 INFO:tasks.rgw_multisite:creating master zonegroup.. 2020-11-09T18:09:26.846 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:09:26.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:09:26.847 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-09T18:09:26.931 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:09:26.932 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:09:26.956 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:09:26.953+0000 7f0d784190c0 -1 old format 2020-11-09T18:09:26.971 INFO:teuthology.orchestra.run.smithi051.stdout:{ 2020-11-09T18:09:26.971 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "test-zonegroup", 2020-11-09T18:09:26.972 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "test-zonegroup", 2020-11-09T18:09:26.972 INFO:teuthology.orchestra.run.smithi051.stdout: "api_name": "", 2020-11-09T18:09:26.972 INFO:teuthology.orchestra.run.smithi051.stdout: "is_master": "true", 2020-11-09T18:09:26.972 INFO:teuthology.orchestra.run.smithi051.stdout: "endpoints": [ 2020-11-09T18:09:26.972 INFO:teuthology.orchestra.run.smithi051.stdout: "http://smithi051.front.sepia.ceph.com:8000" 2020-11-09T18:09:26.972 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:09:26.972 INFO:teuthology.orchestra.run.smithi051.stdout: "hostnames": [], 2020-11-09T18:09:26.973 INFO:teuthology.orchestra.run.smithi051.stdout: "hostnames_s3website": [], 2020-11-09T18:09:26.973 INFO:teuthology.orchestra.run.smithi051.stdout: "master_zone": "", 2020-11-09T18:09:26.973 INFO:teuthology.orchestra.run.smithi051.stdout: "zones": [], 2020-11-09T18:09:26.973 INFO:teuthology.orchestra.run.smithi051.stdout: "placement_targets": [], 2020-11-09T18:09:26.973 INFO:teuthology.orchestra.run.smithi051.stdout: "default_placement": "", 2020-11-09T18:09:26.973 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_id": "0de9c5fb-e243-4abd-b414-59ffbc2cb575", 2020-11-09T18:09:26.973 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_policy": { 2020-11-09T18:09:26.974 INFO:teuthology.orchestra.run.smithi051.stdout: "groups": [] 2020-11-09T18:09:26.974 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:09:26.974 INFO:teuthology.orchestra.run.smithi051.stdout:} 2020-11-09T18:09:26.975 INFO:tasks.rgw_multisite:creating master zone.. 2020-11-09T18:09:26.975 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'ZAW7MMQ493TT1LMM', '--secret', 'HQHAA9hTQSSDkU0yH5qgAQS7JzFsECDH', '--endpoints', 'http://smithi051.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:09:26.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'ZAW7MMQ493TT1LMM', '--secret', 'HQHAA9hTQSSDkU0yH5qgAQS7JzFsECDH', '--endpoints', 'http://smithi051.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:09:26.976 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --master --access-key ZAW7MMQ493TT1LMM --secret HQHAA9hTQSSDkU0yH5qgAQS7JzFsECDH --endpoints http://smithi051.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-09T18:09:27.065 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:09:27.066 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:09:27.142 INFO:teuthology.orchestra.run.smithi051.stdout:{ 2020-11-09T18:09:27.143 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:09:27.143 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "test-zone1", 2020-11-09T18:09:27.143 INFO:teuthology.orchestra.run.smithi051.stdout: "domain_root": "test-zone1.rgw.meta:root", 2020-11-09T18:09:27.143 INFO:teuthology.orchestra.run.smithi051.stdout: "control_pool": "test-zone1.rgw.control", 2020-11-09T18:09:27.143 INFO:teuthology.orchestra.run.smithi051.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2020-11-09T18:09:27.144 INFO:teuthology.orchestra.run.smithi051.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2020-11-09T18:09:27.144 INFO:teuthology.orchestra.run.smithi051.stdout: "log_pool": "test-zone1.rgw.log", 2020-11-09T18:09:27.144 INFO:teuthology.orchestra.run.smithi051.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2020-11-09T18:09:27.144 INFO:teuthology.orchestra.run.smithi051.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2020-11-09T18:09:27.144 INFO:teuthology.orchestra.run.smithi051.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2020-11-09T18:09:27.144 INFO:teuthology.orchestra.run.smithi051.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2020-11-09T18:09:27.145 INFO:teuthology.orchestra.run.smithi051.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2020-11-09T18:09:27.145 INFO:teuthology.orchestra.run.smithi051.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2020-11-09T18:09:27.145 INFO:teuthology.orchestra.run.smithi051.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2020-11-09T18:09:27.145 INFO:teuthology.orchestra.run.smithi051.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2020-11-09T18:09:27.145 INFO:teuthology.orchestra.run.smithi051.stdout: "otp_pool": "test-zone1.rgw.otp", 2020-11-09T18:09:27.145 INFO:teuthology.orchestra.run.smithi051.stdout: "system_key": { 2020-11-09T18:09:27.146 INFO:teuthology.orchestra.run.smithi051.stdout: "access_key": "ZAW7MMQ493TT1LMM", 2020-11-09T18:09:27.146 INFO:teuthology.orchestra.run.smithi051.stdout: "secret_key": "HQHAA9hTQSSDkU0yH5qgAQS7JzFsECDH" 2020-11-09T18:09:27.146 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:09:27.146 INFO:teuthology.orchestra.run.smithi051.stdout: "placement_pools": [ 2020-11-09T18:09:27.146 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:09:27.146 INFO:teuthology.orchestra.run.smithi051.stdout: "key": "default-placement", 2020-11-09T18:09:27.147 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:09:27.147 INFO:teuthology.orchestra.run.smithi051.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2020-11-09T18:09:27.147 INFO:teuthology.orchestra.run.smithi051.stdout: "storage_classes": { 2020-11-09T18:09:27.147 INFO:teuthology.orchestra.run.smithi051.stdout: "STANDARD": { 2020-11-09T18:09:27.147 INFO:teuthology.orchestra.run.smithi051.stdout: "data_pool": "test-zone1.rgw.buckets.data" 2020-11-09T18:09:27.147 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:09:27.148 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:09:27.148 INFO:teuthology.orchestra.run.smithi051.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2020-11-09T18:09:27.148 INFO:teuthology.orchestra.run.smithi051.stdout: "index_type": 0 2020-11-09T18:09:27.148 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:09:27.148 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:09:27.148 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:09:27.149 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_id": "0de9c5fb-e243-4abd-b414-59ffbc2cb575", 2020-11-09T18:09:27.149 INFO:teuthology.orchestra.run.smithi051.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2020-11-09T18:09:27.149 INFO:teuthology.orchestra.run.smithi051.stdout:} 2020-11-09T18:09:27.150 INFO:teuthology.orchestra.run.smithi051:> sudo ceph osd pool create test-zone1.rgw.buckets.data 64 64 --cluster c1 2020-11-09T18:09:28.462 INFO:teuthology.orchestra.run.smithi051.stderr:pool 'test-zone1.rgw.buckets.data' created 2020-11-09T18:09:28.485 INFO:teuthology.orchestra.run.smithi051:> sudo ceph osd pool application enable test-zone1.rgw.buckets.data rgw --cluster c1 2020-11-09T18:09:29.492 INFO:teuthology.orchestra.run.smithi051.stderr:enabled application 'rgw' on pool 'test-zone1.rgw.buckets.data' 2020-11-09T18:09:29.506 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:09:29.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:09:29.507 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-09T18:09:29.547 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:09:29.548 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:09:29.570 INFO:teuthology.orchestra.run.smithi051.stdout:{ 2020-11-09T18:09:29.570 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:09:29.570 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "test-zone1", 2020-11-09T18:09:29.570 INFO:teuthology.orchestra.run.smithi051.stdout: "domain_root": "test-zone1.rgw.meta:root", 2020-11-09T18:09:29.571 INFO:teuthology.orchestra.run.smithi051.stdout: "control_pool": "test-zone1.rgw.control", 2020-11-09T18:09:29.571 INFO:teuthology.orchestra.run.smithi051.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2020-11-09T18:09:29.571 INFO:teuthology.orchestra.run.smithi051.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2020-11-09T18:09:29.571 INFO:teuthology.orchestra.run.smithi051.stdout: "log_pool": "test-zone1.rgw.log", 2020-11-09T18:09:29.571 INFO:teuthology.orchestra.run.smithi051.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2020-11-09T18:09:29.571 INFO:teuthology.orchestra.run.smithi051.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2020-11-09T18:09:29.571 INFO:teuthology.orchestra.run.smithi051.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2020-11-09T18:09:29.572 INFO:teuthology.orchestra.run.smithi051.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2020-11-09T18:09:29.572 INFO:teuthology.orchestra.run.smithi051.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2020-11-09T18:09:29.572 INFO:teuthology.orchestra.run.smithi051.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2020-11-09T18:09:29.572 INFO:teuthology.orchestra.run.smithi051.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2020-11-09T18:09:29.572 INFO:teuthology.orchestra.run.smithi051.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2020-11-09T18:09:29.573 INFO:teuthology.orchestra.run.smithi051.stdout: "otp_pool": "test-zone1.rgw.otp", 2020-11-09T18:09:29.573 INFO:teuthology.orchestra.run.smithi051.stdout: "system_key": { 2020-11-09T18:09:29.573 INFO:teuthology.orchestra.run.smithi051.stdout: "access_key": "ZAW7MMQ493TT1LMM", 2020-11-09T18:09:29.573 INFO:teuthology.orchestra.run.smithi051.stdout: "secret_key": "HQHAA9hTQSSDkU0yH5qgAQS7JzFsECDH" 2020-11-09T18:09:29.574 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:09:29.574 INFO:teuthology.orchestra.run.smithi051.stdout: "placement_pools": [ 2020-11-09T18:09:29.574 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:09:29.574 INFO:teuthology.orchestra.run.smithi051.stdout: "key": "default-placement", 2020-11-09T18:09:29.574 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:09:29.574 INFO:teuthology.orchestra.run.smithi051.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2020-11-09T18:09:29.575 INFO:teuthology.orchestra.run.smithi051.stdout: "storage_classes": { 2020-11-09T18:09:29.575 INFO:teuthology.orchestra.run.smithi051.stdout: "STANDARD": { 2020-11-09T18:09:29.575 INFO:teuthology.orchestra.run.smithi051.stdout: "data_pool": "test-zone1.rgw.buckets.data", 2020-11-09T18:09:29.575 INFO:teuthology.orchestra.run.smithi051.stdout: "compression_type": "random" 2020-11-09T18:09:29.575 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:09:29.575 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:09:29.575 INFO:teuthology.orchestra.run.smithi051.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2020-11-09T18:09:29.575 INFO:teuthology.orchestra.run.smithi051.stdout: "index_type": 0 2020-11-09T18:09:29.576 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:09:29.576 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:09:29.576 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:09:29.576 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_id": "", 2020-11-09T18:09:29.576 INFO:teuthology.orchestra.run.smithi051.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2020-11-09T18:09:29.576 INFO:teuthology.orchestra.run.smithi051.stdout:} 2020-11-09T18:09:29.577 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:09:29.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:09:29.578 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-09T18:09:29.658 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:09:29.659 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:09:37.728 INFO:teuthology.orchestra.run.smithi051.stdout:{ 2020-11-09T18:09:37.728 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:09:37.728 INFO:teuthology.orchestra.run.smithi051.stdout: "epoch": 1, 2020-11-09T18:09:37.728 INFO:teuthology.orchestra.run.smithi051.stdout: "predecessor_uuid": "7de5cbf1-978a-419f-aebf-c2a9e564b43c", 2020-11-09T18:09:37.729 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_status": [], 2020-11-09T18:09:37.729 INFO:teuthology.orchestra.run.smithi051.stdout: "period_map": { 2020-11-09T18:09:37.729 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:09:37.729 INFO:teuthology.orchestra.run.smithi051.stdout: "zonegroups": [ 2020-11-09T18:09:37.729 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:09:37.729 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "test-zonegroup", 2020-11-09T18:09:37.729 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "test-zonegroup", 2020-11-09T18:09:37.730 INFO:teuthology.orchestra.run.smithi051.stdout: "api_name": "", 2020-11-09T18:09:37.730 INFO:teuthology.orchestra.run.smithi051.stdout: "is_master": "true", 2020-11-09T18:09:37.730 INFO:teuthology.orchestra.run.smithi051.stdout: "endpoints": [ 2020-11-09T18:09:37.730 INFO:teuthology.orchestra.run.smithi051.stdout: "http://smithi051.front.sepia.ceph.com:8000" 2020-11-09T18:09:37.730 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:09:37.730 INFO:teuthology.orchestra.run.smithi051.stdout: "hostnames": [], 2020-11-09T18:09:37.731 INFO:teuthology.orchestra.run.smithi051.stdout: "hostnames_s3website": [], 2020-11-09T18:09:37.731 INFO:teuthology.orchestra.run.smithi051.stdout: "master_zone": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:09:37.731 INFO:teuthology.orchestra.run.smithi051.stdout: "zones": [ 2020-11-09T18:09:37.731 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:09:37.731 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:09:37.731 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "test-zone1", 2020-11-09T18:09:37.731 INFO:teuthology.orchestra.run.smithi051.stdout: "endpoints": [ 2020-11-09T18:09:37.732 INFO:teuthology.orchestra.run.smithi051.stdout: "http://smithi051.front.sepia.ceph.com:8000" 2020-11-09T18:09:37.732 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:09:37.732 INFO:teuthology.orchestra.run.smithi051.stdout: "log_meta": "false", 2020-11-09T18:09:37.732 INFO:teuthology.orchestra.run.smithi051.stdout: "log_data": "false", 2020-11-09T18:09:37.732 INFO:teuthology.orchestra.run.smithi051.stdout: "bucket_index_max_shards": 11, 2020-11-09T18:09:37.732 INFO:teuthology.orchestra.run.smithi051.stdout: "read_only": "false", 2020-11-09T18:09:37.732 INFO:teuthology.orchestra.run.smithi051.stdout: "tier_type": "", 2020-11-09T18:09:37.733 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from_all": "true", 2020-11-09T18:09:37.733 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from": [], 2020-11-09T18:09:37.733 INFO:teuthology.orchestra.run.smithi051.stdout: "redirect_zone": "" 2020-11-09T18:09:37.733 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:09:37.734 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:09:37.734 INFO:teuthology.orchestra.run.smithi051.stdout: "placement_targets": [ 2020-11-09T18:09:37.734 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:09:37.734 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "default-placement", 2020-11-09T18:09:37.734 INFO:teuthology.orchestra.run.smithi051.stdout: "tags": [], 2020-11-09T18:09:37.734 INFO:teuthology.orchestra.run.smithi051.stdout: "storage_classes": [ 2020-11-09T18:09:37.734 INFO:teuthology.orchestra.run.smithi051.stdout: "STANDARD" 2020-11-09T18:09:37.734 INFO:teuthology.orchestra.run.smithi051.stdout: ] 2020-11-09T18:09:37.735 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:09:37.735 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:09:37.735 INFO:teuthology.orchestra.run.smithi051.stdout: "default_placement": "default-placement", 2020-11-09T18:09:37.735 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_id": "0de9c5fb-e243-4abd-b414-59ffbc2cb575", 2020-11-09T18:09:37.735 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_policy": { 2020-11-09T18:09:37.735 INFO:teuthology.orchestra.run.smithi051.stdout: "groups": [] 2020-11-09T18:09:37.736 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:09:37.736 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:09:37.736 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:09:37.736 INFO:teuthology.orchestra.run.smithi051.stdout: "short_zone_ids": [ 2020-11-09T18:09:37.736 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:09:37.736 INFO:teuthology.orchestra.run.smithi051.stdout: "key": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:09:37.736 INFO:teuthology.orchestra.run.smithi051.stdout: "val": 2019612975 2020-11-09T18:09:37.736 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:09:37.737 INFO:teuthology.orchestra.run.smithi051.stdout: ] 2020-11-09T18:09:37.737 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:09:37.737 INFO:teuthology.orchestra.run.smithi051.stdout: "master_zonegroup": "test-zonegroup", 2020-11-09T18:09:37.737 INFO:teuthology.orchestra.run.smithi051.stdout: "master_zone": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:09:37.737 INFO:teuthology.orchestra.run.smithi051.stdout: "period_config": { 2020-11-09T18:09:37.737 INFO:teuthology.orchestra.run.smithi051.stdout: "bucket_quota": { 2020-11-09T18:09:37.737 INFO:teuthology.orchestra.run.smithi051.stdout: "enabled": false, 2020-11-09T18:09:37.738 INFO:teuthology.orchestra.run.smithi051.stdout: "check_on_raw": false, 2020-11-09T18:09:37.738 INFO:teuthology.orchestra.run.smithi051.stdout: "max_size": -1, 2020-11-09T18:09:37.738 INFO:teuthology.orchestra.run.smithi051.stdout: "max_size_kb": 0, 2020-11-09T18:09:37.738 INFO:teuthology.orchestra.run.smithi051.stdout: "max_objects": -1 2020-11-09T18:09:37.738 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:09:37.738 INFO:teuthology.orchestra.run.smithi051.stdout: "user_quota": { 2020-11-09T18:09:37.739 INFO:teuthology.orchestra.run.smithi051.stdout: "enabled": false, 2020-11-09T18:09:37.739 INFO:teuthology.orchestra.run.smithi051.stdout: "check_on_raw": false, 2020-11-09T18:09:37.739 INFO:teuthology.orchestra.run.smithi051.stdout: "max_size": -1, 2020-11-09T18:09:37.739 INFO:teuthology.orchestra.run.smithi051.stdout: "max_size_kb": 0, 2020-11-09T18:09:37.739 INFO:teuthology.orchestra.run.smithi051.stdout: "max_objects": -1 2020-11-09T18:09:37.739 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:09:37.739 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:09:37.739 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_id": "0de9c5fb-e243-4abd-b414-59ffbc2cb575", 2020-11-09T18:09:37.740 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_name": "test-realm", 2020-11-09T18:09:37.740 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:09:37.740 INFO:teuthology.orchestra.run.smithi051.stdout:} 2020-11-09T18:09:37.741 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2020-11-09T18:09:37.741 INFO:teuthology.orchestra.run:waiting for 300 2020-11-09T18:09:37.789 INFO:tasks.rgw.c1.client.0.smithi051.stdout:2020-11-09T18:09:37.789+0000 7f6b737fe700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 16345) UID: 0 2020-11-09T18:09:37.789 INFO:tasks.rgw.c1.client.0.smithi051.stdout:2020-11-09T18:09:37.789+0000 7f6b8f567100 -1 shutting down 2020-11-09T18:09:40.120 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2020-11-09T18:09:43.844 INFO:tasks.rgw.c1.client.0:Stopped 2020-11-09T18:09:43.844 INFO:tasks.rgw.c1.client.0:Restarting daemon 2020-11-09T18:09:43.845 INFO:teuthology.orchestra.run.smithi051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2020-11-09T18:09:43.847 INFO:tasks.rgw.c1.client.0:Started 2020-11-09T18:09:43.848 INFO:teuthology.orchestra.run.smithi051:> curl http://smithi051.front.sepia.ceph.com:8000 2020-11-09T18:09:43.863 INFO:teuthology.orchestra.run.smithi051.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-09T18:09:43.864 INFO:teuthology.orchestra.run.smithi051.stderr: Dload Upload Total Spent Left Speed 2020-11-09T18:09:43.867 INFO:teuthology.orchestra.run.smithi051.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi051.front.sepia.ceph.com port 8000: Connection refused 2020-11-09T18:09:43.868 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-11-09T18:09:44.869 INFO:teuthology.orchestra.run.smithi051:> curl http://smithi051.front.sepia.ceph.com:8000 2020-11-09T18:09:44.885 INFO:teuthology.orchestra.run.smithi051.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-09T18:09:44.886 INFO:teuthology.orchestra.run.smithi051.stderr: Dload Upload Total Spent Left Speed 2020-11-09T18:09:44.892 INFO:teuthology.orchestra.run.smithi051.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 35666 0 --:--:-- --:--:-- --:--:-- 35666 2020-11-09T18:09:44.893 INFO:teuthology.orchestra.run.smithi051.stdout:anonymous 2020-11-09T18:09:44.893 INFO:tasks.rgw_multisite:creating admin user.. 2020-11-09T18:09:44.893 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'ZAW7MMQ493TT1LMM', '--secret', 'HQHAA9hTQSSDkU0yH5qgAQS7JzFsECDH', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:09:44.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'ZAW7MMQ493TT1LMM', '--secret', 'HQHAA9hTQSSDkU0yH5qgAQS7JzFsECDH', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:09:44.894 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key ZAW7MMQ493TT1LMM --secret HQHAA9hTQSSDkU0yH5qgAQS7JzFsECDH --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-09T18:09:44.986 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:09:44.987 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:09:45.200 INFO:teuthology.orchestra.run.smithi051.stdout:{ 2020-11-09T18:09:45.200 INFO:teuthology.orchestra.run.smithi051.stdout: "user_id": "realm-admin", 2020-11-09T18:09:45.200 INFO:teuthology.orchestra.run.smithi051.stdout: "display_name": "Realm Admin", 2020-11-09T18:09:45.200 INFO:teuthology.orchestra.run.smithi051.stdout: "email": "", 2020-11-09T18:09:45.201 INFO:teuthology.orchestra.run.smithi051.stdout: "suspended": 0, 2020-11-09T18:09:45.201 INFO:teuthology.orchestra.run.smithi051.stdout: "max_buckets": 1000, 2020-11-09T18:09:45.201 INFO:teuthology.orchestra.run.smithi051.stdout: "subusers": [], 2020-11-09T18:09:45.201 INFO:teuthology.orchestra.run.smithi051.stdout: "keys": [ 2020-11-09T18:09:45.201 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:09:45.201 INFO:teuthology.orchestra.run.smithi051.stdout: "user": "realm-admin", 2020-11-09T18:09:45.201 INFO:teuthology.orchestra.run.smithi051.stdout: "access_key": "ZAW7MMQ493TT1LMM", 2020-11-09T18:09:45.202 INFO:teuthology.orchestra.run.smithi051.stdout: "secret_key": "HQHAA9hTQSSDkU0yH5qgAQS7JzFsECDH" 2020-11-09T18:09:45.202 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:09:45.202 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:09:45.202 INFO:teuthology.orchestra.run.smithi051.stdout: "swift_keys": [], 2020-11-09T18:09:45.202 INFO:teuthology.orchestra.run.smithi051.stdout: "caps": [], 2020-11-09T18:09:45.202 INFO:teuthology.orchestra.run.smithi051.stdout: "op_mask": "read, write, delete", 2020-11-09T18:09:45.202 INFO:teuthology.orchestra.run.smithi051.stdout: "system": "true", 2020-11-09T18:09:45.203 INFO:teuthology.orchestra.run.smithi051.stdout: "default_placement": "", 2020-11-09T18:09:45.203 INFO:teuthology.orchestra.run.smithi051.stdout: "default_storage_class": "", 2020-11-09T18:09:45.203 INFO:teuthology.orchestra.run.smithi051.stdout: "placement_tags": [], 2020-11-09T18:09:45.203 INFO:teuthology.orchestra.run.smithi051.stdout: "bucket_quota": { 2020-11-09T18:09:45.203 INFO:teuthology.orchestra.run.smithi051.stdout: "enabled": false, 2020-11-09T18:09:45.203 INFO:teuthology.orchestra.run.smithi051.stdout: "check_on_raw": false, 2020-11-09T18:09:45.203 INFO:teuthology.orchestra.run.smithi051.stdout: "max_size": -1, 2020-11-09T18:09:45.204 INFO:teuthology.orchestra.run.smithi051.stdout: "max_size_kb": 0, 2020-11-09T18:09:45.204 INFO:teuthology.orchestra.run.smithi051.stdout: "max_objects": -1 2020-11-09T18:09:45.204 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:09:45.204 INFO:teuthology.orchestra.run.smithi051.stdout: "user_quota": { 2020-11-09T18:09:45.204 INFO:teuthology.orchestra.run.smithi051.stdout: "enabled": false, 2020-11-09T18:09:45.204 INFO:teuthology.orchestra.run.smithi051.stdout: "check_on_raw": false, 2020-11-09T18:09:45.204 INFO:teuthology.orchestra.run.smithi051.stdout: "max_size": -1, 2020-11-09T18:09:45.205 INFO:teuthology.orchestra.run.smithi051.stdout: "max_size_kb": 0, 2020-11-09T18:09:45.205 INFO:teuthology.orchestra.run.smithi051.stdout: "max_objects": -1 2020-11-09T18:09:45.205 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:09:45.205 INFO:teuthology.orchestra.run.smithi051.stdout: "temp_url_keys": [], 2020-11-09T18:09:45.205 INFO:teuthology.orchestra.run.smithi051.stdout: "type": "rgw", 2020-11-09T18:09:45.205 INFO:teuthology.orchestra.run.smithi051.stdout: "mfa_ids": [] 2020-11-09T18:09:45.205 INFO:teuthology.orchestra.run.smithi051.stdout:} 2020-11-09T18:09:45.205 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:09:45.229 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2020-11-09T18:09:45.229 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi051.front.sepia.ceph.com:8000', '--access-key', 'ZAW7MMQ493TT1LMM', '--secret', 'HQHAA9hTQSSDkU0yH5qgAQS7JzFsECDH', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:09:45.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi051.front.sepia.ceph.com:8000', '--access-key', 'ZAW7MMQ493TT1LMM', '--secret', 'HQHAA9hTQSSDkU0yH5qgAQS7JzFsECDH', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:09:45.230 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi051.front.sepia.ceph.com:8000 --access-key ZAW7MMQ493TT1LMM --secret HQHAA9hTQSSDkU0yH5qgAQS7JzFsECDH --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-09T18:09:45.272 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:09:45.273 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:09:45.389 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:09:45.390 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "0de9c5fb-e243-4abd-b414-59ffbc2cb575", 2020-11-09T18:09:45.390 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "test-realm", 2020-11-09T18:09:45.390 INFO:teuthology.orchestra.run.smithi078.stdout: "current_period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:09:45.390 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 2 2020-11-09T18:09:45.390 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:09:45.391 INFO:tasks.rgw_multisite:creating zone.. 2020-11-09T18:09:45.392 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'ZAW7MMQ493TT1LMM', '--secret', 'HQHAA9hTQSSDkU0yH5qgAQS7JzFsECDH', '--endpoints', 'http://smithi078.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:09:45.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'ZAW7MMQ493TT1LMM', '--secret', 'HQHAA9hTQSSDkU0yH5qgAQS7JzFsECDH', '--endpoints', 'http://smithi078.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:09:45.392 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --access-key ZAW7MMQ493TT1LMM --secret HQHAA9hTQSSDkU0yH5qgAQS7JzFsECDH --endpoints http://smithi078.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-09T18:09:45.479 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:09:45.480 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:09:45.548 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:45.543+0000 7f16dac6a0c0 0 failed reading obj info from .rgw.root:zone_info.11f516a7-3288-49a6-9a89-b45ac1dae91a: (2) No such file or directory 2020-11-09T18:09:45.549 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:09:45.543+0000 7f16dac6a0c0 0 WARNING: could not read zone params for zone id=11f516a7-3288-49a6-9a89-b45ac1dae91a name=test-zone1 2020-11-09T18:09:45.565 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:09:45.565 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc", 2020-11-09T18:09:45.565 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "test-zone2", 2020-11-09T18:09:45.565 INFO:teuthology.orchestra.run.smithi078.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-11-09T18:09:45.566 INFO:teuthology.orchestra.run.smithi078.stdout: "control_pool": "test-zone2.rgw.control", 2020-11-09T18:09:45.566 INFO:teuthology.orchestra.run.smithi078.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-11-09T18:09:45.566 INFO:teuthology.orchestra.run.smithi078.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-11-09T18:09:45.566 INFO:teuthology.orchestra.run.smithi078.stdout: "log_pool": "test-zone2.rgw.log", 2020-11-09T18:09:45.566 INFO:teuthology.orchestra.run.smithi078.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-11-09T18:09:45.566 INFO:teuthology.orchestra.run.smithi078.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-11-09T18:09:45.567 INFO:teuthology.orchestra.run.smithi078.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-11-09T18:09:45.567 INFO:teuthology.orchestra.run.smithi078.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-11-09T18:09:45.567 INFO:teuthology.orchestra.run.smithi078.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-11-09T18:09:45.567 INFO:teuthology.orchestra.run.smithi078.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-11-09T18:09:45.567 INFO:teuthology.orchestra.run.smithi078.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-11-09T18:09:45.567 INFO:teuthology.orchestra.run.smithi078.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-11-09T18:09:45.567 INFO:teuthology.orchestra.run.smithi078.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-11-09T18:09:45.568 INFO:teuthology.orchestra.run.smithi078.stdout: "system_key": { 2020-11-09T18:09:45.568 INFO:teuthology.orchestra.run.smithi078.stdout: "access_key": "ZAW7MMQ493TT1LMM", 2020-11-09T18:09:45.568 INFO:teuthology.orchestra.run.smithi078.stdout: "secret_key": "HQHAA9hTQSSDkU0yH5qgAQS7JzFsECDH" 2020-11-09T18:09:45.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:09:45.568 INFO:teuthology.orchestra.run.smithi078.stdout: "placement_pools": [ 2020-11-09T18:09:45.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:09:45.569 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "default-placement", 2020-11-09T18:09:45.569 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:09:45.569 INFO:teuthology.orchestra.run.smithi078.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-11-09T18:09:45.569 INFO:teuthology.orchestra.run.smithi078.stdout: "storage_classes": { 2020-11-09T18:09:45.569 INFO:teuthology.orchestra.run.smithi078.stdout: "STANDARD": { 2020-11-09T18:09:45.569 INFO:teuthology.orchestra.run.smithi078.stdout: "data_pool": "test-zone2.rgw.buckets.data" 2020-11-09T18:09:45.569 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:09:45.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:09:45.571 INFO:teuthology.orchestra.run.smithi078.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-11-09T18:09:45.571 INFO:teuthology.orchestra.run.smithi078.stdout: "index_type": 0 2020-11-09T18:09:45.571 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:09:45.571 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:09:45.571 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:09:45.572 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_id": "0de9c5fb-e243-4abd-b414-59ffbc2cb575", 2020-11-09T18:09:45.572 INFO:teuthology.orchestra.run.smithi078.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2020-11-09T18:09:45.572 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:09:45.573 INFO:teuthology.orchestra.run.smithi078:> sudo ceph osd pool create test-zone2.rgw.buckets.data 64 64 --cluster c2 2020-11-09T18:09:45.825 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2020-11-09T18:09:46.510 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'test-zone2.rgw.buckets.data' created 2020-11-09T18:09:46.532 INFO:teuthology.orchestra.run.smithi078:> sudo ceph osd pool application enable test-zone2.rgw.buckets.data rgw --cluster c2 2020-11-09T18:09:47.532 INFO:teuthology.orchestra.run.smithi078.stderr:enabled application 'rgw' on pool 'test-zone2.rgw.buckets.data' 2020-11-09T18:09:47.554 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:09:47.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:09:47.556 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-09T18:09:47.595 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:09:47.595 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:09:47.616 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:09:47.616 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc", 2020-11-09T18:09:47.616 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "test-zone2", 2020-11-09T18:09:47.617 INFO:teuthology.orchestra.run.smithi078.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-11-09T18:09:47.617 INFO:teuthology.orchestra.run.smithi078.stdout: "control_pool": "test-zone2.rgw.control", 2020-11-09T18:09:47.617 INFO:teuthology.orchestra.run.smithi078.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-11-09T18:09:47.617 INFO:teuthology.orchestra.run.smithi078.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-11-09T18:09:47.617 INFO:teuthology.orchestra.run.smithi078.stdout: "log_pool": "test-zone2.rgw.log", 2020-11-09T18:09:47.617 INFO:teuthology.orchestra.run.smithi078.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-11-09T18:09:47.617 INFO:teuthology.orchestra.run.smithi078.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-11-09T18:09:47.618 INFO:teuthology.orchestra.run.smithi078.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-11-09T18:09:47.618 INFO:teuthology.orchestra.run.smithi078.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-11-09T18:09:47.618 INFO:teuthology.orchestra.run.smithi078.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-11-09T18:09:47.618 INFO:teuthology.orchestra.run.smithi078.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-11-09T18:09:47.618 INFO:teuthology.orchestra.run.smithi078.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-11-09T18:09:47.618 INFO:teuthology.orchestra.run.smithi078.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-11-09T18:09:47.618 INFO:teuthology.orchestra.run.smithi078.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-11-09T18:09:47.619 INFO:teuthology.orchestra.run.smithi078.stdout: "system_key": { 2020-11-09T18:09:47.619 INFO:teuthology.orchestra.run.smithi078.stdout: "access_key": "ZAW7MMQ493TT1LMM", 2020-11-09T18:09:47.619 INFO:teuthology.orchestra.run.smithi078.stdout: "secret_key": "HQHAA9hTQSSDkU0yH5qgAQS7JzFsECDH" 2020-11-09T18:09:47.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:09:47.619 INFO:teuthology.orchestra.run.smithi078.stdout: "placement_pools": [ 2020-11-09T18:09:47.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:09:47.619 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "default-placement", 2020-11-09T18:09:47.620 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:09:47.620 INFO:teuthology.orchestra.run.smithi078.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-11-09T18:09:47.620 INFO:teuthology.orchestra.run.smithi078.stdout: "storage_classes": { 2020-11-09T18:09:47.620 INFO:teuthology.orchestra.run.smithi078.stdout: "STANDARD": { 2020-11-09T18:09:47.620 INFO:teuthology.orchestra.run.smithi078.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2020-11-09T18:09:47.620 INFO:teuthology.orchestra.run.smithi078.stdout: "compression_type": "random" 2020-11-09T18:09:47.620 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:09:47.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:09:47.621 INFO:teuthology.orchestra.run.smithi078.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-11-09T18:09:47.621 INFO:teuthology.orchestra.run.smithi078.stdout: "index_type": 0 2020-11-09T18:09:47.621 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:09:47.621 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:09:47.621 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:09:47.621 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_id": "", 2020-11-09T18:09:47.621 INFO:teuthology.orchestra.run.smithi078.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2020-11-09T18:09:47.622 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:09:47.623 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:09:47.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:09:47.623 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-09T18:09:47.702 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:09:47.703 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:09:55.702 INFO:teuthology.orchestra.run.smithi078.stderr:Sending period to new master zone 11f516a7-3288-49a6-9a89-b45ac1dae91a 2020-11-09T18:09:55.799 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:09:55.799 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:09:55.799 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 2, 2020-11-09T18:09:55.799 INFO:teuthology.orchestra.run.smithi078.stdout: "predecessor_uuid": "7de5cbf1-978a-419f-aebf-c2a9e564b43c", 2020-11-09T18:09:55.800 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": [], 2020-11-09T18:09:55.800 INFO:teuthology.orchestra.run.smithi078.stdout: "period_map": { 2020-11-09T18:09:55.800 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:09:55.800 INFO:teuthology.orchestra.run.smithi078.stdout: "zonegroups": [ 2020-11-09T18:09:55.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:09:55.800 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "test-zonegroup", 2020-11-09T18:09:55.800 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "test-zonegroup", 2020-11-09T18:09:55.801 INFO:teuthology.orchestra.run.smithi078.stdout: "api_name": "", 2020-11-09T18:09:55.801 INFO:teuthology.orchestra.run.smithi078.stdout: "is_master": "true", 2020-11-09T18:09:55.801 INFO:teuthology.orchestra.run.smithi078.stdout: "endpoints": [ 2020-11-09T18:09:55.801 INFO:teuthology.orchestra.run.smithi078.stdout: "http://smithi051.front.sepia.ceph.com:8000" 2020-11-09T18:09:55.801 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:09:55.801 INFO:teuthology.orchestra.run.smithi078.stdout: "hostnames": [], 2020-11-09T18:09:55.802 INFO:teuthology.orchestra.run.smithi078.stdout: "hostnames_s3website": [], 2020-11-09T18:09:55.802 INFO:teuthology.orchestra.run.smithi078.stdout: "master_zone": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:09:55.802 INFO:teuthology.orchestra.run.smithi078.stdout: "zones": [ 2020-11-09T18:09:55.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:09:55.802 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:09:55.802 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "test-zone1", 2020-11-09T18:09:55.802 INFO:teuthology.orchestra.run.smithi078.stdout: "endpoints": [ 2020-11-09T18:09:55.803 INFO:teuthology.orchestra.run.smithi078.stdout: "http://smithi051.front.sepia.ceph.com:8000" 2020-11-09T18:09:55.803 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:09:55.803 INFO:teuthology.orchestra.run.smithi078.stdout: "log_meta": "false", 2020-11-09T18:09:55.803 INFO:teuthology.orchestra.run.smithi078.stdout: "log_data": "true", 2020-11-09T18:09:55.803 INFO:teuthology.orchestra.run.smithi078.stdout: "bucket_index_max_shards": 11, 2020-11-09T18:09:55.803 INFO:teuthology.orchestra.run.smithi078.stdout: "read_only": "false", 2020-11-09T18:09:55.803 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_type": "", 2020-11-09T18:09:55.803 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_from_all": "true", 2020-11-09T18:09:55.804 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_from": [], 2020-11-09T18:09:55.804 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect_zone": "" 2020-11-09T18:09:55.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:09:55.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:09:55.804 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc", 2020-11-09T18:09:55.804 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "test-zone2", 2020-11-09T18:09:55.804 INFO:teuthology.orchestra.run.smithi078.stdout: "endpoints": [ 2020-11-09T18:09:55.805 INFO:teuthology.orchestra.run.smithi078.stdout: "http://smithi078.front.sepia.ceph.com:8000" 2020-11-09T18:09:55.805 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:09:55.805 INFO:teuthology.orchestra.run.smithi078.stdout: "log_meta": "false", 2020-11-09T18:09:55.805 INFO:teuthology.orchestra.run.smithi078.stdout: "log_data": "true", 2020-11-09T18:09:55.805 INFO:teuthology.orchestra.run.smithi078.stdout: "bucket_index_max_shards": 11, 2020-11-09T18:09:55.805 INFO:teuthology.orchestra.run.smithi078.stdout: "read_only": "false", 2020-11-09T18:09:55.805 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_type": "", 2020-11-09T18:09:55.806 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_from_all": "true", 2020-11-09T18:09:55.806 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_from": [], 2020-11-09T18:09:55.806 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect_zone": "" 2020-11-09T18:09:55.806 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:09:55.806 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:09:55.806 INFO:teuthology.orchestra.run.smithi078.stdout: "placement_targets": [ 2020-11-09T18:09:55.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:09:55.807 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "default-placement", 2020-11-09T18:09:55.807 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2020-11-09T18:09:55.807 INFO:teuthology.orchestra.run.smithi078.stdout: "storage_classes": [ 2020-11-09T18:09:55.807 INFO:teuthology.orchestra.run.smithi078.stdout: "STANDARD" 2020-11-09T18:09:55.807 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:09:55.807 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:09:55.808 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:09:55.808 INFO:teuthology.orchestra.run.smithi078.stdout: "default_placement": "default-placement", 2020-11-09T18:09:55.808 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_id": "0de9c5fb-e243-4abd-b414-59ffbc2cb575", 2020-11-09T18:09:55.808 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_policy": { 2020-11-09T18:09:55.808 INFO:teuthology.orchestra.run.smithi078.stdout: "groups": [] 2020-11-09T18:09:55.808 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:09:55.808 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:09:55.809 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:09:55.809 INFO:teuthology.orchestra.run.smithi078.stdout: "short_zone_ids": [ 2020-11-09T18:09:55.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:09:55.809 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:09:55.809 INFO:teuthology.orchestra.run.smithi078.stdout: "val": 2019612975 2020-11-09T18:09:55.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:09:55.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:09:55.810 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc", 2020-11-09T18:09:55.810 INFO:teuthology.orchestra.run.smithi078.stdout: "val": 2281443694 2020-11-09T18:09:55.810 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:09:55.810 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:09:55.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:09:55.810 INFO:teuthology.orchestra.run.smithi078.stdout: "master_zonegroup": "test-zonegroup", 2020-11-09T18:09:55.810 INFO:teuthology.orchestra.run.smithi078.stdout: "master_zone": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:09:55.811 INFO:teuthology.orchestra.run.smithi078.stdout: "period_config": { 2020-11-09T18:09:55.811 INFO:teuthology.orchestra.run.smithi078.stdout: "bucket_quota": { 2020-11-09T18:09:55.811 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": false, 2020-11-09T18:09:55.811 INFO:teuthology.orchestra.run.smithi078.stdout: "check_on_raw": false, 2020-11-09T18:09:55.811 INFO:teuthology.orchestra.run.smithi078.stdout: "max_size": -1, 2020-11-09T18:09:55.811 INFO:teuthology.orchestra.run.smithi078.stdout: "max_size_kb": 0, 2020-11-09T18:09:55.811 INFO:teuthology.orchestra.run.smithi078.stdout: "max_objects": -1 2020-11-09T18:09:55.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:09:55.812 INFO:teuthology.orchestra.run.smithi078.stdout: "user_quota": { 2020-11-09T18:09:55.812 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": false, 2020-11-09T18:09:55.812 INFO:teuthology.orchestra.run.smithi078.stdout: "check_on_raw": false, 2020-11-09T18:09:55.812 INFO:teuthology.orchestra.run.smithi078.stdout: "max_size": -1, 2020-11-09T18:09:55.812 INFO:teuthology.orchestra.run.smithi078.stdout: "max_size_kb": 0, 2020-11-09T18:09:55.812 INFO:teuthology.orchestra.run.smithi078.stdout: "max_objects": -1 2020-11-09T18:09:55.813 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:09:55.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:09:55.813 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_id": "0de9c5fb-e243-4abd-b414-59ffbc2cb575", 2020-11-09T18:09:55.813 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_name": "test-realm", 2020-11-09T18:09:55.813 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:09:55.813 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:09:55.814 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2020-11-09T18:09:55.815 INFO:teuthology.orchestra.run:waiting for 300 2020-11-09T18:09:55.857 INFO:tasks.rgw.c2.client.0.smithi078.stdout:2020-11-09T18:09:55.855+0000 7f0080923700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 16625) UID: 0 2020-11-09T18:09:55.857 INFO:tasks.rgw.c2.client.0.smithi078.stdout:2020-11-09T18:09:55.855+0000 7f00983ba100 -1 shutting down 2020-11-09T18:09:58.737 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2020-11-09T18:10:01.917 INFO:tasks.rgw.c2.client.0:Stopped 2020-11-09T18:10:01.917 INFO:tasks.rgw.c2.client.0:Restarting daemon 2020-11-09T18:10:01.918 INFO:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2020-11-09T18:10:01.920 INFO:tasks.rgw.c2.client.0:Started 2020-11-09T18:10:01.920 INFO:teuthology.orchestra.run.smithi078:> curl http://smithi078.front.sepia.ceph.com:8000 2020-11-09T18:10:01.938 INFO:teuthology.orchestra.run.smithi078.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-09T18:10:01.938 INFO:teuthology.orchestra.run.smithi078.stderr: Dload Upload Total Spent Left Speed 2020-11-09T18:10:01.942 INFO:teuthology.orchestra.run.smithi078.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi078.front.sepia.ceph.com port 8000: Connection refused 2020-11-09T18:10:01.943 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-11-09T18:10:02.944 INFO:teuthology.orchestra.run.smithi078:> curl http://smithi078.front.sepia.ceph.com:8000 2020-11-09T18:10:02.960 INFO:teuthology.orchestra.run.smithi078.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-09T18:10:02.960 INFO:teuthology.orchestra.run.smithi078.stderr: Dload Upload Total Spent Left Speed 2020-11-09T18:10:02.966 INFO:teuthology.orchestra.run.smithi078.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 30571 0 --:--:-- --:--:-- --:--:-- 35666 2020-11-09T18:10:02.967 INFO:teuthology.orchestra.run.smithi078.stdout:anonymous 2020-11-09T18:10:02.968 INFO:tasks.rgw_multisite:creating zone.. 2020-11-09T18:10:02.968 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'ZAW7MMQ493TT1LMM', '--secret', 'HQHAA9hTQSSDkU0yH5qgAQS7JzFsECDH', '--endpoints', 'http://smithi051.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:10:02.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'ZAW7MMQ493TT1LMM', '--secret', 'HQHAA9hTQSSDkU0yH5qgAQS7JzFsECDH', '--endpoints', 'http://smithi051.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:10:02.969 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key ZAW7MMQ493TT1LMM --secret HQHAA9hTQSSDkU0yH5qgAQS7JzFsECDH --endpoints http://smithi051.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-09T18:10:03.016 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:10:03.016 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:10:03.081 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:10:03.081+0000 7f99cce950c0 0 failed reading obj info from .rgw.root:zone_info.4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc: (2) No such file or directory 2020-11-09T18:10:03.081 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:10:03.081+0000 7f99cce950c0 0 WARNING: could not read zone params for zone id=4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc name=test-zone2 2020-11-09T18:10:03.093 INFO:teuthology.orchestra.run.smithi051.stdout:{ 2020-11-09T18:10:03.093 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6", 2020-11-09T18:10:03.093 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "test-zone3", 2020-11-09T18:10:03.094 INFO:teuthology.orchestra.run.smithi051.stdout: "domain_root": "test-zone3.rgw.meta:root", 2020-11-09T18:10:03.094 INFO:teuthology.orchestra.run.smithi051.stdout: "control_pool": "test-zone3.rgw.control", 2020-11-09T18:10:03.094 INFO:teuthology.orchestra.run.smithi051.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2020-11-09T18:10:03.094 INFO:teuthology.orchestra.run.smithi051.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2020-11-09T18:10:03.094 INFO:teuthology.orchestra.run.smithi051.stdout: "log_pool": "test-zone3.rgw.log", 2020-11-09T18:10:03.094 INFO:teuthology.orchestra.run.smithi051.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2020-11-09T18:10:03.095 INFO:teuthology.orchestra.run.smithi051.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2020-11-09T18:10:03.095 INFO:teuthology.orchestra.run.smithi051.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2020-11-09T18:10:03.095 INFO:teuthology.orchestra.run.smithi051.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2020-11-09T18:10:03.095 INFO:teuthology.orchestra.run.smithi051.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2020-11-09T18:10:03.095 INFO:teuthology.orchestra.run.smithi051.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2020-11-09T18:10:03.095 INFO:teuthology.orchestra.run.smithi051.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2020-11-09T18:10:03.095 INFO:teuthology.orchestra.run.smithi051.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2020-11-09T18:10:03.096 INFO:teuthology.orchestra.run.smithi051.stdout: "otp_pool": "test-zone3.rgw.otp", 2020-11-09T18:10:03.096 INFO:teuthology.orchestra.run.smithi051.stdout: "system_key": { 2020-11-09T18:10:03.096 INFO:teuthology.orchestra.run.smithi051.stdout: "access_key": "ZAW7MMQ493TT1LMM", 2020-11-09T18:10:03.096 INFO:teuthology.orchestra.run.smithi051.stdout: "secret_key": "HQHAA9hTQSSDkU0yH5qgAQS7JzFsECDH" 2020-11-09T18:10:03.096 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:10:03.096 INFO:teuthology.orchestra.run.smithi051.stdout: "placement_pools": [ 2020-11-09T18:10:03.096 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:10:03.097 INFO:teuthology.orchestra.run.smithi051.stdout: "key": "default-placement", 2020-11-09T18:10:03.097 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:10:03.097 INFO:teuthology.orchestra.run.smithi051.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2020-11-09T18:10:03.097 INFO:teuthology.orchestra.run.smithi051.stdout: "storage_classes": { 2020-11-09T18:10:03.097 INFO:teuthology.orchestra.run.smithi051.stdout: "STANDARD": { 2020-11-09T18:10:03.097 INFO:teuthology.orchestra.run.smithi051.stdout: "data_pool": "test-zone3.rgw.buckets.data" 2020-11-09T18:10:03.097 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:10:03.097 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:10:03.098 INFO:teuthology.orchestra.run.smithi051.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2020-11-09T18:10:03.098 INFO:teuthology.orchestra.run.smithi051.stdout: "index_type": 0 2020-11-09T18:10:03.098 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:10:03.098 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:10:03.098 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:10:03.098 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_id": "0de9c5fb-e243-4abd-b414-59ffbc2cb575", 2020-11-09T18:10:03.098 INFO:teuthology.orchestra.run.smithi051.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2020-11-09T18:10:03.099 INFO:teuthology.orchestra.run.smithi051.stdout:} 2020-11-09T18:10:03.099 INFO:teuthology.orchestra.run.smithi051:> sudo ceph osd pool create test-zone3.rgw.buckets.data 64 64 --cluster c1 2020-11-09T18:10:04.450 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2020-11-09T18:10:04.474 INFO:teuthology.orchestra.run.smithi051.stderr:pool 'test-zone3.rgw.buckets.data' created 2020-11-09T18:10:04.501 INFO:teuthology.orchestra.run.smithi051:> sudo ceph osd pool application enable test-zone3.rgw.buckets.data rgw --cluster c1 2020-11-09T18:10:05.513 INFO:teuthology.orchestra.run.smithi051.stderr:enabled application 'rgw' on pool 'test-zone3.rgw.buckets.data' 2020-11-09T18:10:05.538 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:10:05.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:10:05.539 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-09T18:10:05.584 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:10:05.584 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:10:05.608 INFO:teuthology.orchestra.run.smithi051.stdout:{ 2020-11-09T18:10:05.609 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6", 2020-11-09T18:10:05.609 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "test-zone3", 2020-11-09T18:10:05.609 INFO:teuthology.orchestra.run.smithi051.stdout: "domain_root": "test-zone3.rgw.meta:root", 2020-11-09T18:10:05.610 INFO:teuthology.orchestra.run.smithi051.stdout: "control_pool": "test-zone3.rgw.control", 2020-11-09T18:10:05.610 INFO:teuthology.orchestra.run.smithi051.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2020-11-09T18:10:05.610 INFO:teuthology.orchestra.run.smithi051.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2020-11-09T18:10:05.610 INFO:teuthology.orchestra.run.smithi051.stdout: "log_pool": "test-zone3.rgw.log", 2020-11-09T18:10:05.610 INFO:teuthology.orchestra.run.smithi051.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2020-11-09T18:10:05.611 INFO:teuthology.orchestra.run.smithi051.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2020-11-09T18:10:05.611 INFO:teuthology.orchestra.run.smithi051.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2020-11-09T18:10:05.611 INFO:teuthology.orchestra.run.smithi051.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2020-11-09T18:10:05.611 INFO:teuthology.orchestra.run.smithi051.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2020-11-09T18:10:05.611 INFO:teuthology.orchestra.run.smithi051.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2020-11-09T18:10:05.612 INFO:teuthology.orchestra.run.smithi051.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2020-11-09T18:10:05.612 INFO:teuthology.orchestra.run.smithi051.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2020-11-09T18:10:05.612 INFO:teuthology.orchestra.run.smithi051.stdout: "otp_pool": "test-zone3.rgw.otp", 2020-11-09T18:10:05.612 INFO:teuthology.orchestra.run.smithi051.stdout: "system_key": { 2020-11-09T18:10:05.612 INFO:teuthology.orchestra.run.smithi051.stdout: "access_key": "ZAW7MMQ493TT1LMM", 2020-11-09T18:10:05.613 INFO:teuthology.orchestra.run.smithi051.stdout: "secret_key": "HQHAA9hTQSSDkU0yH5qgAQS7JzFsECDH" 2020-11-09T18:10:05.613 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:10:05.613 INFO:teuthology.orchestra.run.smithi051.stdout: "placement_pools": [ 2020-11-09T18:10:05.613 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:10:05.613 INFO:teuthology.orchestra.run.smithi051.stdout: "key": "default-placement", 2020-11-09T18:10:05.613 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:10:05.614 INFO:teuthology.orchestra.run.smithi051.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2020-11-09T18:10:05.614 INFO:teuthology.orchestra.run.smithi051.stdout: "storage_classes": { 2020-11-09T18:10:05.614 INFO:teuthology.orchestra.run.smithi051.stdout: "STANDARD": { 2020-11-09T18:10:05.615 INFO:teuthology.orchestra.run.smithi051.stdout: "data_pool": "test-zone3.rgw.buckets.data", 2020-11-09T18:10:05.615 INFO:teuthology.orchestra.run.smithi051.stdout: "compression_type": "random" 2020-11-09T18:10:05.615 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:10:05.615 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:10:05.616 INFO:teuthology.orchestra.run.smithi051.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2020-11-09T18:10:05.616 INFO:teuthology.orchestra.run.smithi051.stdout: "index_type": 0 2020-11-09T18:10:05.616 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:10:05.617 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:10:05.617 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:10:05.617 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_id": "", 2020-11-09T18:10:05.618 INFO:teuthology.orchestra.run.smithi051.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2020-11-09T18:10:05.618 INFO:teuthology.orchestra.run.smithi051.stdout:} 2020-11-09T18:10:05.619 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:10:05.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:10:05.621 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-09T18:10:05.703 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:10:05.704 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:10:13.725 INFO:teuthology.orchestra.run.smithi051.stderr:Sending period to new master zone 11f516a7-3288-49a6-9a89-b45ac1dae91a 2020-11-09T18:10:13.790 INFO:teuthology.orchestra.run.smithi051.stdout:{ 2020-11-09T18:10:13.791 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:10:13.791 INFO:teuthology.orchestra.run.smithi051.stdout: "epoch": 3, 2020-11-09T18:10:13.791 INFO:teuthology.orchestra.run.smithi051.stdout: "predecessor_uuid": "7de5cbf1-978a-419f-aebf-c2a9e564b43c", 2020-11-09T18:10:13.791 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_status": [], 2020-11-09T18:10:13.792 INFO:teuthology.orchestra.run.smithi051.stdout: "period_map": { 2020-11-09T18:10:13.792 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:10:13.792 INFO:teuthology.orchestra.run.smithi051.stdout: "zonegroups": [ 2020-11-09T18:10:13.792 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:10:13.792 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "test-zonegroup", 2020-11-09T18:10:13.793 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "test-zonegroup", 2020-11-09T18:10:13.793 INFO:teuthology.orchestra.run.smithi051.stdout: "api_name": "", 2020-11-09T18:10:13.793 INFO:teuthology.orchestra.run.smithi051.stdout: "is_master": "true", 2020-11-09T18:10:13.793 INFO:teuthology.orchestra.run.smithi051.stdout: "endpoints": [ 2020-11-09T18:10:13.793 INFO:teuthology.orchestra.run.smithi051.stdout: "http://smithi051.front.sepia.ceph.com:8000" 2020-11-09T18:10:13.794 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:10:13.794 INFO:teuthology.orchestra.run.smithi051.stdout: "hostnames": [], 2020-11-09T18:10:13.794 INFO:teuthology.orchestra.run.smithi051.stdout: "hostnames_s3website": [], 2020-11-09T18:10:13.794 INFO:teuthology.orchestra.run.smithi051.stdout: "master_zone": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:10:13.794 INFO:teuthology.orchestra.run.smithi051.stdout: "zones": [ 2020-11-09T18:10:13.795 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:10:13.795 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:10:13.795 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "test-zone1", 2020-11-09T18:10:13.795 INFO:teuthology.orchestra.run.smithi051.stdout: "endpoints": [ 2020-11-09T18:10:13.796 INFO:teuthology.orchestra.run.smithi051.stdout: "http://smithi051.front.sepia.ceph.com:8000" 2020-11-09T18:10:13.796 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:10:13.796 INFO:teuthology.orchestra.run.smithi051.stdout: "log_meta": "false", 2020-11-09T18:10:13.796 INFO:teuthology.orchestra.run.smithi051.stdout: "log_data": "true", 2020-11-09T18:10:13.797 INFO:teuthology.orchestra.run.smithi051.stdout: "bucket_index_max_shards": 11, 2020-11-09T18:10:13.797 INFO:teuthology.orchestra.run.smithi051.stdout: "read_only": "false", 2020-11-09T18:10:13.797 INFO:teuthology.orchestra.run.smithi051.stdout: "tier_type": "", 2020-11-09T18:10:13.797 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from_all": "true", 2020-11-09T18:10:13.798 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from": [], 2020-11-09T18:10:13.798 INFO:teuthology.orchestra.run.smithi051.stdout: "redirect_zone": "" 2020-11-09T18:10:13.798 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:10:13.799 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:10:13.799 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc", 2020-11-09T18:10:13.799 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "test-zone2", 2020-11-09T18:10:13.799 INFO:teuthology.orchestra.run.smithi051.stdout: "endpoints": [ 2020-11-09T18:10:13.800 INFO:teuthology.orchestra.run.smithi051.stdout: "http://smithi078.front.sepia.ceph.com:8000" 2020-11-09T18:10:13.800 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:10:13.800 INFO:teuthology.orchestra.run.smithi051.stdout: "log_meta": "false", 2020-11-09T18:10:13.800 INFO:teuthology.orchestra.run.smithi051.stdout: "log_data": "true", 2020-11-09T18:10:13.800 INFO:teuthology.orchestra.run.smithi051.stdout: "bucket_index_max_shards": 11, 2020-11-09T18:10:13.801 INFO:teuthology.orchestra.run.smithi051.stdout: "read_only": "false", 2020-11-09T18:10:13.801 INFO:teuthology.orchestra.run.smithi051.stdout: "tier_type": "", 2020-11-09T18:10:13.801 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from_all": "true", 2020-11-09T18:10:13.802 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from": [], 2020-11-09T18:10:13.802 INFO:teuthology.orchestra.run.smithi051.stdout: "redirect_zone": "" 2020-11-09T18:10:13.802 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:10:13.803 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:10:13.803 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6", 2020-11-09T18:10:13.803 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "test-zone3", 2020-11-09T18:10:13.803 INFO:teuthology.orchestra.run.smithi051.stdout: "endpoints": [ 2020-11-09T18:10:13.803 INFO:teuthology.orchestra.run.smithi051.stdout: "http://smithi051.front.sepia.ceph.com:8001" 2020-11-09T18:10:13.803 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:10:13.804 INFO:teuthology.orchestra.run.smithi051.stdout: "log_meta": "false", 2020-11-09T18:10:13.804 INFO:teuthology.orchestra.run.smithi051.stdout: "log_data": "true", 2020-11-09T18:10:13.804 INFO:teuthology.orchestra.run.smithi051.stdout: "bucket_index_max_shards": 11, 2020-11-09T18:10:13.804 INFO:teuthology.orchestra.run.smithi051.stdout: "read_only": "false", 2020-11-09T18:10:13.805 INFO:teuthology.orchestra.run.smithi051.stdout: "tier_type": "", 2020-11-09T18:10:13.805 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from_all": "true", 2020-11-09T18:10:13.805 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from": [], 2020-11-09T18:10:13.805 INFO:teuthology.orchestra.run.smithi051.stdout: "redirect_zone": "" 2020-11-09T18:10:13.806 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:10:13.806 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:10:13.806 INFO:teuthology.orchestra.run.smithi051.stdout: "placement_targets": [ 2020-11-09T18:10:13.806 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:10:13.807 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "default-placement", 2020-11-09T18:10:13.807 INFO:teuthology.orchestra.run.smithi051.stdout: "tags": [], 2020-11-09T18:10:13.808 INFO:teuthology.orchestra.run.smithi051.stdout: "storage_classes": [ 2020-11-09T18:10:13.808 INFO:teuthology.orchestra.run.smithi051.stdout: "STANDARD" 2020-11-09T18:10:13.808 INFO:teuthology.orchestra.run.smithi051.stdout: ] 2020-11-09T18:10:13.808 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:10:13.809 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:10:13.809 INFO:teuthology.orchestra.run.smithi051.stdout: "default_placement": "default-placement", 2020-11-09T18:10:13.809 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_id": "0de9c5fb-e243-4abd-b414-59ffbc2cb575", 2020-11-09T18:10:13.810 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_policy": { 2020-11-09T18:10:13.810 INFO:teuthology.orchestra.run.smithi051.stdout: "groups": [] 2020-11-09T18:10:13.810 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:10:13.810 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:10:13.811 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:10:13.811 INFO:teuthology.orchestra.run.smithi051.stdout: "short_zone_ids": [ 2020-11-09T18:10:13.811 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:10:13.811 INFO:teuthology.orchestra.run.smithi051.stdout: "key": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:10:13.812 INFO:teuthology.orchestra.run.smithi051.stdout: "val": 2019612975 2020-11-09T18:10:13.812 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:10:13.812 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:10:13.812 INFO:teuthology.orchestra.run.smithi051.stdout: "key": "4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc", 2020-11-09T18:10:13.813 INFO:teuthology.orchestra.run.smithi051.stdout: "val": 2281443694 2020-11-09T18:10:13.813 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:10:13.813 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:10:13.814 INFO:teuthology.orchestra.run.smithi051.stdout: "key": "a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6", 2020-11-09T18:10:13.815 INFO:teuthology.orchestra.run.smithi051.stdout: "val": 648291153 2020-11-09T18:10:13.816 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:10:13.816 INFO:teuthology.orchestra.run.smithi051.stdout: ] 2020-11-09T18:10:13.816 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:10:13.816 INFO:teuthology.orchestra.run.smithi051.stdout: "master_zonegroup": "test-zonegroup", 2020-11-09T18:10:13.816 INFO:teuthology.orchestra.run.smithi051.stdout: "master_zone": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:10:13.816 INFO:teuthology.orchestra.run.smithi051.stdout: "period_config": { 2020-11-09T18:10:13.816 INFO:teuthology.orchestra.run.smithi051.stdout: "bucket_quota": { 2020-11-09T18:10:13.817 INFO:teuthology.orchestra.run.smithi051.stdout: "enabled": false, 2020-11-09T18:10:13.817 INFO:teuthology.orchestra.run.smithi051.stdout: "check_on_raw": false, 2020-11-09T18:10:13.817 INFO:teuthology.orchestra.run.smithi051.stdout: "max_size": -1, 2020-11-09T18:10:13.817 INFO:teuthology.orchestra.run.smithi051.stdout: "max_size_kb": 0, 2020-11-09T18:10:13.817 INFO:teuthology.orchestra.run.smithi051.stdout: "max_objects": -1 2020-11-09T18:10:13.817 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:10:13.817 INFO:teuthology.orchestra.run.smithi051.stdout: "user_quota": { 2020-11-09T18:10:13.818 INFO:teuthology.orchestra.run.smithi051.stdout: "enabled": false, 2020-11-09T18:10:13.818 INFO:teuthology.orchestra.run.smithi051.stdout: "check_on_raw": false, 2020-11-09T18:10:13.818 INFO:teuthology.orchestra.run.smithi051.stdout: "max_size": -1, 2020-11-09T18:10:13.818 INFO:teuthology.orchestra.run.smithi051.stdout: "max_size_kb": 0, 2020-11-09T18:10:13.819 INFO:teuthology.orchestra.run.smithi051.stdout: "max_objects": -1 2020-11-09T18:10:13.819 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:10:13.819 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:10:13.819 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_id": "0de9c5fb-e243-4abd-b414-59ffbc2cb575", 2020-11-09T18:10:13.819 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_name": "test-realm", 2020-11-09T18:10:13.819 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:10:13.820 INFO:teuthology.orchestra.run.smithi051.stdout:} 2020-11-09T18:10:13.820 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2020-11-09T18:10:13.821 INFO:teuthology.orchestra.run:waiting for 300 2020-11-09T18:10:13.864 INFO:tasks.rgw.c1.client.1.smithi051.stdout:2020-11-09T18:10:13.865+0000 7f9148faf700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 16347) UID: 0 2020-11-09T18:10:13.865 INFO:tasks.rgw.c1.client.1.smithi051.stdout:2020-11-09T18:10:13.865+0000 7f9160245100 -1 shutting down 2020-11-09T18:10:14.256 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2020-11-09T18:10:19.861 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~6s 2020-11-09T18:10:19.923 INFO:tasks.rgw.c1.client.1:Stopped 2020-11-09T18:10:19.924 INFO:tasks.rgw.c1.client.1:Restarting daemon 2020-11-09T18:10:19.924 INFO:teuthology.orchestra.run.smithi051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2020-11-09T18:10:19.927 INFO:tasks.rgw.c1.client.1:Started 2020-11-09T18:10:19.927 INFO:teuthology.orchestra.run.smithi051:> curl http://smithi051.front.sepia.ceph.com:8001 2020-11-09T18:10:19.943 INFO:teuthology.orchestra.run.smithi051.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-09T18:10:19.943 INFO:teuthology.orchestra.run.smithi051.stderr: Dload Upload Total Spent Left Speed 2020-11-09T18:10:19.947 INFO:teuthology.orchestra.run.smithi051.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi051.front.sepia.ceph.com port 8001: Connection refused 2020-11-09T18:10:19.948 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-11-09T18:10:20.950 INFO:teuthology.orchestra.run.smithi051:> curl http://smithi051.front.sepia.ceph.com:8001 2020-11-09T18:10:20.970 INFO:teuthology.orchestra.run.smithi051.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-09T18:10:20.971 INFO:teuthology.orchestra.run.smithi051.stderr: Dload Upload Total Spent Left Speed 2020-11-09T18:10:20.978 INFO:teuthology.orchestra.run.smithi051.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 35666 0 --:--:-- --:--:-- --:--:-- 35666 2020-11-09T18:10:20.979 INFO:teuthology.orchestra.run.smithi051.stdout:anonymous 2020-11-09T18:10:20.979 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2020-11-09T18:10:20.980 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi051.front.sepia.ceph.com:8000', '--access-key', 'ZAW7MMQ493TT1LMM', '--secret', 'HQHAA9hTQSSDkU0yH5qgAQS7JzFsECDH', '--url', 'http://smithi051.front.sepia.ceph.com:8000', '--access-key', 'ZAW7MMQ493TT1LMM', '--secret', 'HQHAA9hTQSSDkU0yH5qgAQS7JzFsECDH', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:10:20.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi051.front.sepia.ceph.com:8000', '--access-key', 'ZAW7MMQ493TT1LMM', '--secret', 'HQHAA9hTQSSDkU0yH5qgAQS7JzFsECDH', '--url', 'http://smithi051.front.sepia.ceph.com:8000', '--access-key', 'ZAW7MMQ493TT1LMM', '--secret', 'HQHAA9hTQSSDkU0yH5qgAQS7JzFsECDH', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:10:20.983 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi051.front.sepia.ceph.com:8000 --access-key ZAW7MMQ493TT1LMM --secret HQHAA9hTQSSDkU0yH5qgAQS7JzFsECDH --url http://smithi051.front.sepia.ceph.com:8000 --access-key ZAW7MMQ493TT1LMM --secret HQHAA9hTQSSDkU0yH5qgAQS7JzFsECDH --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-09T18:10:21.050 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:10:21.051 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:10:21.094 INFO:tasks.rgw.c2.client.0.smithi078.stdout:2020-11-09T18:10:21.091+0000 7fbfc4610700 -1 rgw realm watcher: RGWRealmWatcher::handle_error oid=realms.0de9c5fb-e243-4abd-b414-59ffbc2cb575.control err=-107 2020-11-09T18:10:21.118 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:10:21.119 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "0de9c5fb-e243-4abd-b414-59ffbc2cb575", 2020-11-09T18:10:21.119 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "test-realm", 2020-11-09T18:10:21.119 INFO:teuthology.orchestra.run.smithi078.stdout: "current_period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:10:21.119 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 2 2020-11-09T18:10:21.119 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:10:21.120 INFO:tasks.rgw_multisite:creating zone.. 2020-11-09T18:10:21.121 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'ZAW7MMQ493TT1LMM', '--secret', 'HQHAA9hTQSSDkU0yH5qgAQS7JzFsECDH', '--endpoints', 'http://smithi078.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:10:21.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'ZAW7MMQ493TT1LMM', '--secret', 'HQHAA9hTQSSDkU0yH5qgAQS7JzFsECDH', '--endpoints', 'http://smithi078.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:10:21.122 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key ZAW7MMQ493TT1LMM --secret HQHAA9hTQSSDkU0yH5qgAQS7JzFsECDH --endpoints http://smithi078.front.sepia.ceph.com:8001 --tier-type pubsub --sync-from-all=0 --sync-from test-zone1 --tier-config start_with_full_sync=false,event_retention_days=7 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-09T18:10:21.183 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:10:21.184 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:10:21.247 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:10:21.243+0000 7fcef3e640c0 0 failed reading obj info from .rgw.root:zone_info.11f516a7-3288-49a6-9a89-b45ac1dae91a: (2) No such file or directory 2020-11-09T18:10:21.247 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:10:21.243+0000 7fcef3e640c0 0 WARNING: could not read zone params for zone id=11f516a7-3288-49a6-9a89-b45ac1dae91a name=test-zone1 2020-11-09T18:10:21.249 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:10:21.247+0000 7fcef3e640c0 0 failed reading obj info from .rgw.root:zone_info.a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6: (2) No such file or directory 2020-11-09T18:10:21.249 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:10:21.247+0000 7fcef3e640c0 0 WARNING: could not read zone params for zone id=a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6 name=test-zone3 2020-11-09T18:10:21.258 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:10:21.258 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "4600c141-233a-40b8-a0db-e3e3a34df6c3", 2020-11-09T18:10:21.258 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "test-zone4", 2020-11-09T18:10:21.258 INFO:teuthology.orchestra.run.smithi078.stdout: "domain_root": "test-zone4.rgw.meta:root", 2020-11-09T18:10:21.259 INFO:teuthology.orchestra.run.smithi078.stdout: "control_pool": "test-zone4.rgw.control", 2020-11-09T18:10:21.259 INFO:teuthology.orchestra.run.smithi078.stdout: "gc_pool": "test-zone4.rgw.log:gc", 2020-11-09T18:10:21.259 INFO:teuthology.orchestra.run.smithi078.stdout: "lc_pool": "test-zone4.rgw.log:lc", 2020-11-09T18:10:21.259 INFO:teuthology.orchestra.run.smithi078.stdout: "log_pool": "test-zone4.rgw.log", 2020-11-09T18:10:21.259 INFO:teuthology.orchestra.run.smithi078.stdout: "intent_log_pool": "test-zone4.rgw.log:intent", 2020-11-09T18:10:21.259 INFO:teuthology.orchestra.run.smithi078.stdout: "usage_log_pool": "test-zone4.rgw.log:usage", 2020-11-09T18:10:21.260 INFO:teuthology.orchestra.run.smithi078.stdout: "roles_pool": "test-zone4.rgw.meta:roles", 2020-11-09T18:10:21.260 INFO:teuthology.orchestra.run.smithi078.stdout: "reshard_pool": "test-zone4.rgw.log:reshard", 2020-11-09T18:10:21.260 INFO:teuthology.orchestra.run.smithi078.stdout: "user_keys_pool": "test-zone4.rgw.meta:users.keys", 2020-11-09T18:10:21.260 INFO:teuthology.orchestra.run.smithi078.stdout: "user_email_pool": "test-zone4.rgw.meta:users.email", 2020-11-09T18:10:21.260 INFO:teuthology.orchestra.run.smithi078.stdout: "user_swift_pool": "test-zone4.rgw.meta:users.swift", 2020-11-09T18:10:21.260 INFO:teuthology.orchestra.run.smithi078.stdout: "user_uid_pool": "test-zone4.rgw.meta:users.uid", 2020-11-09T18:10:21.260 INFO:teuthology.orchestra.run.smithi078.stdout: "otp_pool": "test-zone4.rgw.otp", 2020-11-09T18:10:21.261 INFO:teuthology.orchestra.run.smithi078.stdout: "system_key": { 2020-11-09T18:10:21.261 INFO:teuthology.orchestra.run.smithi078.stdout: "access_key": "ZAW7MMQ493TT1LMM", 2020-11-09T18:10:21.261 INFO:teuthology.orchestra.run.smithi078.stdout: "secret_key": "HQHAA9hTQSSDkU0yH5qgAQS7JzFsECDH" 2020-11-09T18:10:21.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:10:21.261 INFO:teuthology.orchestra.run.smithi078.stdout: "placement_pools": [ 2020-11-09T18:10:21.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:10:21.261 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "default-placement", 2020-11-09T18:10:21.261 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:10:21.262 INFO:teuthology.orchestra.run.smithi078.stdout: "index_pool": "test-zone4.rgw.buckets.index", 2020-11-09T18:10:21.262 INFO:teuthology.orchestra.run.smithi078.stdout: "storage_classes": { 2020-11-09T18:10:21.262 INFO:teuthology.orchestra.run.smithi078.stdout: "STANDARD": { 2020-11-09T18:10:21.262 INFO:teuthology.orchestra.run.smithi078.stdout: "data_pool": "test-zone4.rgw.buckets.data" 2020-11-09T18:10:21.262 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:10:21.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:10:21.262 INFO:teuthology.orchestra.run.smithi078.stdout: "data_extra_pool": "test-zone4.rgw.buckets.non-ec", 2020-11-09T18:10:21.263 INFO:teuthology.orchestra.run.smithi078.stdout: "index_type": 0 2020-11-09T18:10:21.263 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:10:21.263 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:10:21.263 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:10:21.263 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_config": { 2020-11-09T18:10:21.263 INFO:teuthology.orchestra.run.smithi078.stdout: "event_retention_days": 7, 2020-11-09T18:10:21.263 INFO:teuthology.orchestra.run.smithi078.stdout: "start_with_full_sync": false 2020-11-09T18:10:21.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:10:21.264 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_id": "0de9c5fb-e243-4abd-b414-59ffbc2cb575", 2020-11-09T18:10:21.264 INFO:teuthology.orchestra.run.smithi078.stdout: "notif_pool": "test-zone4.rgw.log:notif" 2020-11-09T18:10:21.264 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:10:21.266 INFO:teuthology.orchestra.run.smithi078:> sudo ceph osd pool create test-zone4.rgw.buckets.data 64 64 --cluster c2 2020-11-09T18:10:22.502 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'test-zone4.rgw.buckets.data' created 2020-11-09T18:10:22.529 INFO:teuthology.orchestra.run.smithi078:> sudo ceph osd pool application enable test-zone4.rgw.buckets.data rgw --cluster c2 2020-11-09T18:10:23.548 INFO:teuthology.orchestra.run.smithi078.stderr:enabled application 'rgw' on pool 'test-zone4.rgw.buckets.data' 2020-11-09T18:10:23.575 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:10:23.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:10:23.576 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-09T18:10:23.610 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:10:23.611 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:10:23.635 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:10:23.636 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "4600c141-233a-40b8-a0db-e3e3a34df6c3", 2020-11-09T18:10:23.636 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "test-zone4", 2020-11-09T18:10:23.636 INFO:teuthology.orchestra.run.smithi078.stdout: "domain_root": "test-zone4.rgw.meta:root", 2020-11-09T18:10:23.636 INFO:teuthology.orchestra.run.smithi078.stdout: "control_pool": "test-zone4.rgw.control", 2020-11-09T18:10:23.636 INFO:teuthology.orchestra.run.smithi078.stdout: "gc_pool": "test-zone4.rgw.log:gc", 2020-11-09T18:10:23.637 INFO:teuthology.orchestra.run.smithi078.stdout: "lc_pool": "test-zone4.rgw.log:lc", 2020-11-09T18:10:23.637 INFO:teuthology.orchestra.run.smithi078.stdout: "log_pool": "test-zone4.rgw.log", 2020-11-09T18:10:23.637 INFO:teuthology.orchestra.run.smithi078.stdout: "intent_log_pool": "test-zone4.rgw.log:intent", 2020-11-09T18:10:23.637 INFO:teuthology.orchestra.run.smithi078.stdout: "usage_log_pool": "test-zone4.rgw.log:usage", 2020-11-09T18:10:23.637 INFO:teuthology.orchestra.run.smithi078.stdout: "roles_pool": "test-zone4.rgw.meta:roles", 2020-11-09T18:10:23.637 INFO:teuthology.orchestra.run.smithi078.stdout: "reshard_pool": "test-zone4.rgw.log:reshard", 2020-11-09T18:10:23.638 INFO:teuthology.orchestra.run.smithi078.stdout: "user_keys_pool": "test-zone4.rgw.meta:users.keys", 2020-11-09T18:10:23.638 INFO:teuthology.orchestra.run.smithi078.stdout: "user_email_pool": "test-zone4.rgw.meta:users.email", 2020-11-09T18:10:23.638 INFO:teuthology.orchestra.run.smithi078.stdout: "user_swift_pool": "test-zone4.rgw.meta:users.swift", 2020-11-09T18:10:23.638 INFO:teuthology.orchestra.run.smithi078.stdout: "user_uid_pool": "test-zone4.rgw.meta:users.uid", 2020-11-09T18:10:23.638 INFO:teuthology.orchestra.run.smithi078.stdout: "otp_pool": "test-zone4.rgw.otp", 2020-11-09T18:10:23.638 INFO:teuthology.orchestra.run.smithi078.stdout: "system_key": { 2020-11-09T18:10:23.639 INFO:teuthology.orchestra.run.smithi078.stdout: "access_key": "ZAW7MMQ493TT1LMM", 2020-11-09T18:10:23.639 INFO:teuthology.orchestra.run.smithi078.stdout: "secret_key": "HQHAA9hTQSSDkU0yH5qgAQS7JzFsECDH" 2020-11-09T18:10:23.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:10:23.639 INFO:teuthology.orchestra.run.smithi078.stdout: "placement_pools": [ 2020-11-09T18:10:23.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:10:23.639 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "default-placement", 2020-11-09T18:10:23.639 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:10:23.640 INFO:teuthology.orchestra.run.smithi078.stdout: "index_pool": "test-zone4.rgw.buckets.index", 2020-11-09T18:10:23.640 INFO:teuthology.orchestra.run.smithi078.stdout: "storage_classes": { 2020-11-09T18:10:23.640 INFO:teuthology.orchestra.run.smithi078.stdout: "STANDARD": { 2020-11-09T18:10:23.640 INFO:teuthology.orchestra.run.smithi078.stdout: "data_pool": "test-zone4.rgw.buckets.data", 2020-11-09T18:10:23.640 INFO:teuthology.orchestra.run.smithi078.stdout: "compression_type": "random" 2020-11-09T18:10:23.640 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:10:23.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:10:23.641 INFO:teuthology.orchestra.run.smithi078.stdout: "data_extra_pool": "test-zone4.rgw.buckets.non-ec", 2020-11-09T18:10:23.641 INFO:teuthology.orchestra.run.smithi078.stdout: "index_type": 0 2020-11-09T18:10:23.641 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:10:23.641 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:10:23.642 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:10:23.642 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_config": { 2020-11-09T18:10:23.642 INFO:teuthology.orchestra.run.smithi078.stdout: "event_retention_days": 7, 2020-11-09T18:10:23.642 INFO:teuthology.orchestra.run.smithi078.stdout: "start_with_full_sync": false 2020-11-09T18:10:23.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:10:23.642 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_id": "", 2020-11-09T18:10:23.642 INFO:teuthology.orchestra.run.smithi078.stdout: "notif_pool": "test-zone4.rgw.log:notif" 2020-11-09T18:10:23.643 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:10:23.644 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:10:23.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:10:23.645 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-09T18:10:23.724 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:10:23.724 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:10:25.568 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2020-11-09T18:10:31.740 INFO:teuthology.orchestra.run.smithi078.stderr:Sending period to new master zone 11f516a7-3288-49a6-9a89-b45ac1dae91a 2020-11-09T18:10:31.818 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:10:31.818 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:10:31.818 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 4, 2020-11-09T18:10:31.819 INFO:teuthology.orchestra.run.smithi078.stdout: "predecessor_uuid": "7de5cbf1-978a-419f-aebf-c2a9e564b43c", 2020-11-09T18:10:31.819 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": [], 2020-11-09T18:10:31.819 INFO:teuthology.orchestra.run.smithi078.stdout: "period_map": { 2020-11-09T18:10:31.819 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:10:31.819 INFO:teuthology.orchestra.run.smithi078.stdout: "zonegroups": [ 2020-11-09T18:10:31.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:10:31.820 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "test-zonegroup", 2020-11-09T18:10:31.820 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "test-zonegroup", 2020-11-09T18:10:31.820 INFO:teuthology.orchestra.run.smithi078.stdout: "api_name": "", 2020-11-09T18:10:31.820 INFO:teuthology.orchestra.run.smithi078.stdout: "is_master": "true", 2020-11-09T18:10:31.820 INFO:teuthology.orchestra.run.smithi078.stdout: "endpoints": [ 2020-11-09T18:10:31.820 INFO:teuthology.orchestra.run.smithi078.stdout: "http://smithi051.front.sepia.ceph.com:8000" 2020-11-09T18:10:31.820 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:10:31.821 INFO:teuthology.orchestra.run.smithi078.stdout: "hostnames": [], 2020-11-09T18:10:31.821 INFO:teuthology.orchestra.run.smithi078.stdout: "hostnames_s3website": [], 2020-11-09T18:10:31.821 INFO:teuthology.orchestra.run.smithi078.stdout: "master_zone": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:10:31.821 INFO:teuthology.orchestra.run.smithi078.stdout: "zones": [ 2020-11-09T18:10:31.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:10:31.821 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:10:31.821 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "test-zone1", 2020-11-09T18:10:31.822 INFO:teuthology.orchestra.run.smithi078.stdout: "endpoints": [ 2020-11-09T18:10:31.822 INFO:teuthology.orchestra.run.smithi078.stdout: "http://smithi051.front.sepia.ceph.com:8000" 2020-11-09T18:10:31.822 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:10:31.822 INFO:teuthology.orchestra.run.smithi078.stdout: "log_meta": "false", 2020-11-09T18:10:31.822 INFO:teuthology.orchestra.run.smithi078.stdout: "log_data": "true", 2020-11-09T18:10:31.822 INFO:teuthology.orchestra.run.smithi078.stdout: "bucket_index_max_shards": 11, 2020-11-09T18:10:31.822 INFO:teuthology.orchestra.run.smithi078.stdout: "read_only": "false", 2020-11-09T18:10:31.823 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_type": "", 2020-11-09T18:10:31.823 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_from_all": "true", 2020-11-09T18:10:31.823 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_from": [], 2020-11-09T18:10:31.823 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect_zone": "" 2020-11-09T18:10:31.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:10:31.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:10:31.824 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "4600c141-233a-40b8-a0db-e3e3a34df6c3", 2020-11-09T18:10:31.824 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "test-zone4", 2020-11-09T18:10:31.824 INFO:teuthology.orchestra.run.smithi078.stdout: "endpoints": [ 2020-11-09T18:10:31.824 INFO:teuthology.orchestra.run.smithi078.stdout: "http://smithi078.front.sepia.ceph.com:8001" 2020-11-09T18:10:31.824 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:10:31.824 INFO:teuthology.orchestra.run.smithi078.stdout: "log_meta": "false", 2020-11-09T18:10:31.824 INFO:teuthology.orchestra.run.smithi078.stdout: "log_data": "true", 2020-11-09T18:10:31.825 INFO:teuthology.orchestra.run.smithi078.stdout: "bucket_index_max_shards": 11, 2020-11-09T18:10:31.825 INFO:teuthology.orchestra.run.smithi078.stdout: "read_only": "false", 2020-11-09T18:10:31.825 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_type": "pubsub", 2020-11-09T18:10:31.825 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_from_all": "false", 2020-11-09T18:10:31.825 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_from": [ 2020-11-09T18:10:31.825 INFO:teuthology.orchestra.run.smithi078.stdout: "test-zone1" 2020-11-09T18:10:31.825 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:10:31.826 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect_zone": "" 2020-11-09T18:10:31.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:10:31.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:10:31.826 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc", 2020-11-09T18:10:31.826 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "test-zone2", 2020-11-09T18:10:31.826 INFO:teuthology.orchestra.run.smithi078.stdout: "endpoints": [ 2020-11-09T18:10:31.827 INFO:teuthology.orchestra.run.smithi078.stdout: "http://smithi078.front.sepia.ceph.com:8000" 2020-11-09T18:10:31.827 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:10:31.827 INFO:teuthology.orchestra.run.smithi078.stdout: "log_meta": "false", 2020-11-09T18:10:31.827 INFO:teuthology.orchestra.run.smithi078.stdout: "log_data": "true", 2020-11-09T18:10:31.827 INFO:teuthology.orchestra.run.smithi078.stdout: "bucket_index_max_shards": 11, 2020-11-09T18:10:31.827 INFO:teuthology.orchestra.run.smithi078.stdout: "read_only": "false", 2020-11-09T18:10:31.827 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_type": "", 2020-11-09T18:10:31.828 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_from_all": "true", 2020-11-09T18:10:31.828 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_from": [], 2020-11-09T18:10:31.828 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect_zone": "" 2020-11-09T18:10:31.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:10:31.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:10:31.828 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6", 2020-11-09T18:10:31.828 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "test-zone3", 2020-11-09T18:10:31.829 INFO:teuthology.orchestra.run.smithi078.stdout: "endpoints": [ 2020-11-09T18:10:31.829 INFO:teuthology.orchestra.run.smithi078.stdout: "http://smithi051.front.sepia.ceph.com:8001" 2020-11-09T18:10:31.829 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:10:31.829 INFO:teuthology.orchestra.run.smithi078.stdout: "log_meta": "false", 2020-11-09T18:10:31.829 INFO:teuthology.orchestra.run.smithi078.stdout: "log_data": "true", 2020-11-09T18:10:31.829 INFO:teuthology.orchestra.run.smithi078.stdout: "bucket_index_max_shards": 11, 2020-11-09T18:10:31.829 INFO:teuthology.orchestra.run.smithi078.stdout: "read_only": "false", 2020-11-09T18:10:31.829 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_type": "", 2020-11-09T18:10:31.830 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_from_all": "true", 2020-11-09T18:10:31.830 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_from": [], 2020-11-09T18:10:31.830 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect_zone": "" 2020-11-09T18:10:31.830 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:10:31.830 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:10:31.830 INFO:teuthology.orchestra.run.smithi078.stdout: "placement_targets": [ 2020-11-09T18:10:31.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:10:31.831 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "default-placement", 2020-11-09T18:10:31.831 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2020-11-09T18:10:31.831 INFO:teuthology.orchestra.run.smithi078.stdout: "storage_classes": [ 2020-11-09T18:10:31.831 INFO:teuthology.orchestra.run.smithi078.stdout: "STANDARD" 2020-11-09T18:10:31.831 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:10:31.831 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:10:31.832 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:10:31.832 INFO:teuthology.orchestra.run.smithi078.stdout: "default_placement": "default-placement", 2020-11-09T18:10:31.832 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_id": "0de9c5fb-e243-4abd-b414-59ffbc2cb575", 2020-11-09T18:10:31.832 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_policy": { 2020-11-09T18:10:31.832 INFO:teuthology.orchestra.run.smithi078.stdout: "groups": [] 2020-11-09T18:10:31.832 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:10:31.833 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:10:31.833 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:10:31.833 INFO:teuthology.orchestra.run.smithi078.stdout: "short_zone_ids": [ 2020-11-09T18:10:31.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:10:31.985 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:10:31.985 INFO:teuthology.orchestra.run.smithi078.stdout: "val": 2019612975 2020-11-09T18:10:31.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:10:31.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:10:31.986 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "4600c141-233a-40b8-a0db-e3e3a34df6c3", 2020-11-09T18:10:31.986 INFO:teuthology.orchestra.run.smithi078.stdout: "val": 1720005997 2020-11-09T18:10:31.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:10:31.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:10:31.986 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc", 2020-11-09T18:10:31.986 INFO:teuthology.orchestra.run.smithi078.stdout: "val": 2281443694 2020-11-09T18:10:31.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:10:31.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:10:31.987 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6", 2020-11-09T18:10:31.987 INFO:teuthology.orchestra.run.smithi078.stdout: "val": 648291153 2020-11-09T18:10:31.987 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:10:31.987 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:10:31.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:10:31.987 INFO:teuthology.orchestra.run.smithi078.stdout: "master_zonegroup": "test-zonegroup", 2020-11-09T18:10:31.988 INFO:teuthology.orchestra.run.smithi078.stdout: "master_zone": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:10:31.988 INFO:teuthology.orchestra.run.smithi078.stdout: "period_config": { 2020-11-09T18:10:31.988 INFO:teuthology.orchestra.run.smithi078.stdout: "bucket_quota": { 2020-11-09T18:10:31.988 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": false, 2020-11-09T18:10:31.988 INFO:teuthology.orchestra.run.smithi078.stdout: "check_on_raw": false, 2020-11-09T18:10:31.988 INFO:teuthology.orchestra.run.smithi078.stdout: "max_size": -1, 2020-11-09T18:10:31.989 INFO:teuthology.orchestra.run.smithi078.stdout: "max_size_kb": 0, 2020-11-09T18:10:31.989 INFO:teuthology.orchestra.run.smithi078.stdout: "max_objects": -1 2020-11-09T18:10:31.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:10:31.989 INFO:teuthology.orchestra.run.smithi078.stdout: "user_quota": { 2020-11-09T18:10:31.990 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": false, 2020-11-09T18:10:31.990 INFO:teuthology.orchestra.run.smithi078.stdout: "check_on_raw": false, 2020-11-09T18:10:31.990 INFO:teuthology.orchestra.run.smithi078.stdout: "max_size": -1, 2020-11-09T18:10:31.990 INFO:teuthology.orchestra.run.smithi078.stdout: "max_size_kb": 0, 2020-11-09T18:10:31.990 INFO:teuthology.orchestra.run.smithi078.stdout: "max_objects": -1 2020-11-09T18:10:31.990 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:10:31.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:10:31.991 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_id": "0de9c5fb-e243-4abd-b414-59ffbc2cb575", 2020-11-09T18:10:31.991 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_name": "test-realm", 2020-11-09T18:10:31.991 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:10:31.991 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:10:31.992 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2020-11-09T18:10:31.992 INFO:teuthology.orchestra.run:waiting for 300 2020-11-09T18:10:32.036 INFO:tasks.rgw.c2.client.1.smithi078.stdout:2020-11-09T18:10:32.035+0000 7f2c42334700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 16627) UID: 0 2020-11-09T18:10:32.036 INFO:tasks.rgw.c2.client.1.smithi078.stdout:2020-11-09T18:10:32.035+0000 7f2c59dcb100 -1 shutting down 2020-11-09T18:10:32.877 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2020-11-09T18:10:38.094 INFO:tasks.rgw.c2.client.1:Stopped 2020-11-09T18:10:38.095 INFO:tasks.rgw.c2.client.1:Restarting daemon 2020-11-09T18:10:38.095 INFO:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2020-11-09T18:10:38.097 INFO:tasks.rgw.c2.client.1:Started 2020-11-09T18:10:38.098 INFO:teuthology.orchestra.run.smithi078:> curl http://smithi078.front.sepia.ceph.com:8001 2020-11-09T18:10:38.114 INFO:teuthology.orchestra.run.smithi078.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-09T18:10:38.114 INFO:teuthology.orchestra.run.smithi078.stderr: Dload Upload Total Spent Left Speed 2020-11-09T18:10:38.118 INFO:teuthology.orchestra.run.smithi078.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi078.front.sepia.ceph.com port 8001: Connection refused 2020-11-09T18:10:38.119 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-11-09T18:10:38.583 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2020-11-09T18:10:39.120 INFO:teuthology.orchestra.run.smithi078:> curl http://smithi078.front.sepia.ceph.com:8001 2020-11-09T18:10:39.138 INFO:teuthology.orchestra.run.smithi078.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-09T18:10:39.139 INFO:teuthology.orchestra.run.smithi078.stderr: Dload Upload Total Spent Left Speed 2020-11-09T18:10:39.143 INFO:teuthology.orchestra.run.smithi078.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 134 100 134 0 0 33500 0 --:--:-- --:--:-- --:--:-- 33500 2020-11-09T18:10:39.143 INFO:teuthology.orchestra.run.smithi078.stdout:{"Code":"MethodNotAllowed","RequestId":"tx000000000000000000001-005fa9861f-10e8-test-zone4","HostId":"10e8-test-zone4-test-zonegroup"} 2020-11-09T18:10:39.144 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2020-11-09T18:10:39.144 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2020-11-09T18:10:39.223 INFO:tasks.rgw_multisite_tests:creating test user.. 2020-11-09T18:10:39.224 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:10:39.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:10:39.224 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-09T18:10:39.269 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:10:39.270 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:10:39.498 INFO:teuthology.orchestra.run.smithi051.stdout:{ 2020-11-09T18:10:39.498 INFO:teuthology.orchestra.run.smithi051.stdout: "user_id": "rgw-multisite-test-user", 2020-11-09T18:10:39.498 INFO:teuthology.orchestra.run.smithi051.stdout: "display_name": "Multisite Test User", 2020-11-09T18:10:39.498 INFO:teuthology.orchestra.run.smithi051.stdout: "email": "", 2020-11-09T18:10:39.498 INFO:teuthology.orchestra.run.smithi051.stdout: "suspended": 0, 2020-11-09T18:10:39.499 INFO:teuthology.orchestra.run.smithi051.stdout: "max_buckets": 1000, 2020-11-09T18:10:39.499 INFO:teuthology.orchestra.run.smithi051.stdout: "subusers": [], 2020-11-09T18:10:39.499 INFO:teuthology.orchestra.run.smithi051.stdout: "keys": [ 2020-11-09T18:10:39.499 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:10:39.499 INFO:teuthology.orchestra.run.smithi051.stdout: "user": "rgw-multisite-test-user", 2020-11-09T18:10:39.499 INFO:teuthology.orchestra.run.smithi051.stdout: "access_key": "3Y1EGES0PAVG4HGR0HEF", 2020-11-09T18:10:39.500 INFO:teuthology.orchestra.run.smithi051.stdout: "secret_key": "R5b2MKLIWBJbfoOLVgRv1BqQewi4Zv49AYTPbjsw" 2020-11-09T18:10:39.500 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:10:39.500 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:10:39.500 INFO:teuthology.orchestra.run.smithi051.stdout: "swift_keys": [], 2020-11-09T18:10:39.500 INFO:teuthology.orchestra.run.smithi051.stdout: "caps": [], 2020-11-09T18:10:39.500 INFO:teuthology.orchestra.run.smithi051.stdout: "op_mask": "read, write, delete", 2020-11-09T18:10:39.500 INFO:teuthology.orchestra.run.smithi051.stdout: "default_placement": "", 2020-11-09T18:10:39.500 INFO:teuthology.orchestra.run.smithi051.stdout: "default_storage_class": "", 2020-11-09T18:10:39.501 INFO:teuthology.orchestra.run.smithi051.stdout: "placement_tags": [], 2020-11-09T18:10:39.501 INFO:teuthology.orchestra.run.smithi051.stdout: "bucket_quota": { 2020-11-09T18:10:39.501 INFO:teuthology.orchestra.run.smithi051.stdout: "enabled": false, 2020-11-09T18:10:39.501 INFO:teuthology.orchestra.run.smithi051.stdout: "check_on_raw": false, 2020-11-09T18:10:39.501 INFO:teuthology.orchestra.run.smithi051.stdout: "max_size": -1, 2020-11-09T18:10:39.501 INFO:teuthology.orchestra.run.smithi051.stdout: "max_size_kb": 0, 2020-11-09T18:10:39.501 INFO:teuthology.orchestra.run.smithi051.stdout: "max_objects": -1 2020-11-09T18:10:39.502 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:10:39.502 INFO:teuthology.orchestra.run.smithi051.stdout: "user_quota": { 2020-11-09T18:10:39.502 INFO:teuthology.orchestra.run.smithi051.stdout: "enabled": false, 2020-11-09T18:10:39.502 INFO:teuthology.orchestra.run.smithi051.stdout: "check_on_raw": false, 2020-11-09T18:10:39.502 INFO:teuthology.orchestra.run.smithi051.stdout: "max_size": -1, 2020-11-09T18:10:39.502 INFO:teuthology.orchestra.run.smithi051.stdout: "max_size_kb": 0, 2020-11-09T18:10:39.502 INFO:teuthology.orchestra.run.smithi051.stdout: "max_objects": -1 2020-11-09T18:10:39.504 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:10:39.504 INFO:teuthology.orchestra.run.smithi051.stdout: "temp_url_keys": [], 2020-11-09T18:10:39.504 INFO:teuthology.orchestra.run.smithi051.stdout: "type": "rgw", 2020-11-09T18:10:39.504 INFO:teuthology.orchestra.run.smithi051.stdout: "mfa_ids": [] 2020-11-09T18:10:39.504 INFO:teuthology.orchestra.run.smithi051.stdout:} 2020-11-09T18:10:39.504 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:10:39.529 INFO:rgw_multi.tests:meta checkpoint 2020-11-09T18:10:39.530 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:10:39.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:10:39.531 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:10:39.608 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:10:39.609 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:10:39.826 INFO:teuthology.orchestra.run.smithi051.stderr:No --period given, using current period=a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9 2020-11-09T18:10:39.858 INFO:teuthology.orchestra.run.smithi051.stdout:[ 2020-11-09T18:10:39.858 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:10:39.859 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945439.492888_50.1", 2020-11-09T18:10:39.859 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "2020-11-09T18:10:39.492888Z" 2020-11-09T18:10:39.859 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:10:39.859 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:10:39.859 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:10:39.860 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:10:39.860 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:10:39.860 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:10:39.860 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:10:39.860 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:10:39.861 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:10:39.861 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:10:39.861 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:10:39.861 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:10:39.861 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:10:39.861 INFO:teuthology.orchestra.run.smithi051.stdout:] 2020-11-09T18:10:39.862 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-09T18:10:39.863 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:10:39.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:10:39.864 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:10:39.904 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:10:39.905 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:10:40.162 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:10:40.163 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:10:40.163 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:10:40.163 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:10:40.163 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:10:40.163 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:10:40.164 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:10:40.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:10:40.164 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:10:40.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:10:40.164 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:10:40.165 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:10:40.165 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:10:40.165 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:10:40.165 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:10:40.165 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:10:40.165 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:10:40.165 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:10:40.166 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:10:40.166 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:10:40.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:10:40.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:10:40.166 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:10:40.166 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:10:40.166 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:10:40.167 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:10:40.167 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:10:40.167 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:10:40.167 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:10:40.167 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:10:40.167 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:10:40.167 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:10:40.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:10:40.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:10:40.168 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:10:40.168 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:10:40.168 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:10:40.168 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:10:40.168 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:10:40.168 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:10:40.169 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:10:40.169 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:10:40.169 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:10:40.169 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:10:40.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:10:40.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:10:40.169 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:10:40.170 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:10:40.170 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:10:40.170 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:10:40.170 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:10:40.170 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:10:40.170 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:10:40.170 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:10:40.171 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:10:40.171 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:10:40.171 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:10:40.171 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:10:40.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:10:40.171 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:10:40.171 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:10:40.171 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:10:40.172 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:10:40.172 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:10:40.173 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945439.492888_50.1 target= 2020-11-09T18:10:45.173 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:10:45.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:10:45.175 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:10:45.216 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:10:45.217 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:10:45.470 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:10:45.470 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:10:45.470 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:10:45.471 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:10:45.471 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:10:45.471 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:10:45.471 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:10:45.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:10:45.471 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:10:45.472 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:10:45.472 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:10:45.472 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:10:45.472 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:10:45.472 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:10:45.472 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:10:45.472 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:10:45.472 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:10:45.473 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:10:45.473 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:10:45.473 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:10:45.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:10:45.473 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:10:45.473 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:10:45.473 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:10:45.474 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:10:45.474 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:10:45.474 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:10:45.474 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:10:45.474 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:10:45.474 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:10:45.474 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:10:45.475 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:10:45.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:10:45.475 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:10:45.475 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:10:45.475 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:10:45.475 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:10:45.475 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:10:45.476 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:10:45.476 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:10:45.476 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:10:45.476 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:10:45.476 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:10:45.476 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:10:45.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:10:45.476 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:10:45.477 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:10:45.477 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:10:45.477 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:10:45.477 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:10:45.477 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:10:45.477 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:10:45.477 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:10:45.478 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:10:45.478 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:10:45.478 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:10:45.478 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:10:45.478 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:10:45.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:10:45.478 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:10:45.479 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:10:45.479 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:10:45.479 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:10:45.479 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:10:45.480 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945439.492888_50.1 target= 2020-11-09T18:10:50.481 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:10:50.482 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:10:50.482 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:10:50.528 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:10:50.529 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:10:50.820 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:10:50.820 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:10:50.821 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:10:50.821 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:10:50.821 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:10:50.821 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:10:50.821 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:10:50.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:10:50.821 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:10:50.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:10:50.822 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:10:50.822 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:10:50.822 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:10:50.822 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:10:50.822 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:10:50.822 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:10:50.823 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:10:50.823 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:10:50.823 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:10:50.823 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:10:50.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:10:50.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:10:50.823 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:10:50.824 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:10:50.824 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:10:50.824 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:10:50.824 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:10:50.824 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:10:50.824 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:10:50.824 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:10:50.824 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:10:50.825 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:10:50.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:10:50.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:10:50.825 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:10:50.825 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:10:50.825 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:10:50.825 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:10:50.826 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:10:50.826 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:10:50.826 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:10:50.826 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:10:50.826 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:10:50.826 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:10:50.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:10:50.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:10:50.827 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:10:50.827 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:10:50.827 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:10:50.827 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:10:50.828 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:10:50.828 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:10:50.828 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:10:50.828 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:10:50.828 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:10:50.828 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:10:50.828 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:10:50.829 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:10:50.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:10:50.829 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:10:50.829 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:10:50.829 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:10:50.829 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:10:50.829 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:10:50.830 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945439.492888_50.1 target= 2020-11-09T18:10:55.832 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:10:55.832 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:10:55.833 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:10:55.873 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:10:55.873 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:10:56.148 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:10:56.148 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:10:56.149 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:10:56.149 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:10:56.149 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:10:56.149 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:10:56.149 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:10:56.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:10:56.150 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:10:56.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:10:56.150 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:10:56.150 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:10:56.150 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:10:56.150 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:10:56.151 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:10:56.151 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:10:56.151 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:10:56.151 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:10:56.151 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:10:56.151 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:10:56.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:10:56.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:10:56.152 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:10:56.152 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:10:56.152 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:10:56.153 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:10:56.153 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:10:56.153 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:10:56.153 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:10:56.153 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:10:56.153 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:10:56.154 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:10:56.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:10:56.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:10:56.154 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:10:56.154 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:10:56.155 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:10:56.155 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:10:56.155 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:10:56.155 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:10:56.155 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:10:56.155 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:10:56.155 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:10:56.156 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:10:56.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:10:56.156 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:10:56.156 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:10:56.156 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:10:56.156 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:10:56.157 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:10:56.157 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:10:56.157 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:10:56.157 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:10:56.158 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:10:56.158 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:10:56.158 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:10:56.158 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:10:56.158 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:10:56.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:10:56.159 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:10:56.159 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:10:56.159 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:10:56.159 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:10:56.159 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:10:56.160 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945439.492888_50.1 target= 2020-11-09T18:11:01.161 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:11:01.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:11:01.163 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:11:01.203 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:11:01.204 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:11:01.441 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:11:01.442 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:11:01.442 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:11:01.442 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:11:01.442 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:11:01.442 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:11:01.442 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:01.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:01.443 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:11:01.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:11:01.443 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:11:01.443 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:11:01.443 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:11:01.443 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:11:01.444 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:11:01.444 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:11:01.444 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:11:01.444 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:11:01.444 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:01.444 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:01.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:01.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:11:01.445 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:11:01.445 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:11:01.445 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:11:01.445 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:11:01.445 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:11:01.445 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:11:01.445 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:11:01.446 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:11:01.446 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:01.446 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:01.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:01.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:11:01.446 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:11:01.446 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:11:01.447 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:11:01.447 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:11:01.447 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:11:01.447 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:11:01.447 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:11:01.447 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:11:01.447 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:01.447 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:01.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:01.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:11:01.448 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:11:01.448 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:11:01.448 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:11:01.448 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:11:01.448 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:11:01.449 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:11:01.449 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:11:01.449 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:11:01.449 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:01.449 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:01.449 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:01.449 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:11:01.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:01.450 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:11:01.450 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:11:01.450 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:11:01.450 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:01.450 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:11:01.451 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945439.492888_50.1 target= 2020-11-09T18:11:06.453 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:11:06.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:11:06.454 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:11:06.497 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:11:06.497 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:11:06.751 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:11:06.752 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:11:06.752 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:11:06.752 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:11:06.752 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:11:06.752 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:11:06.752 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:06.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:06.753 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:11:06.753 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:11:06.753 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:11:06.753 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:11:06.753 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:11:06.753 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:11:06.754 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:11:06.754 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:11:06.754 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:11:06.754 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:11:06.754 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:06.754 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:06.754 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:06.755 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:11:06.755 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:11:06.755 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:11:06.755 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:11:06.755 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:11:06.755 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:11:06.755 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:11:06.756 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:11:06.756 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:11:06.756 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:06.756 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:06.756 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:06.756 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:11:06.756 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:11:06.756 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:11:06.757 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:11:06.757 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:11:06.757 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:11:06.757 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:11:06.757 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:11:06.757 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:11:06.757 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:06.758 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:06.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:06.758 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:11:06.758 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:11:06.758 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:11:06.758 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:11:06.758 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:11:06.759 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:11:06.759 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:11:06.759 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:11:06.759 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:11:06.759 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:06.759 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:06.759 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:06.759 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:11:06.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:06.760 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:11:06.760 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:11:06.760 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:11:06.760 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:06.760 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:11:06.761 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945439.492888_50.1 target= 2020-11-09T18:11:11.763 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:11:11.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:11:11.764 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:11:11.804 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:11:11.805 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:11:12.059 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:11:12.059 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:11:12.059 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:11:12.059 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:11:12.060 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:11:12.060 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:11:12.060 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:12.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:12.060 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:11:12.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:11:12.061 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:11:12.061 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:11:12.061 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:11:12.061 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:11:12.061 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:11:12.061 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:11:12.061 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:11:12.062 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:11:12.062 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:12.062 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:12.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:12.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:11:12.062 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:11:12.063 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:11:12.063 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:11:12.063 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:11:12.063 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:11:12.063 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:11:12.063 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:11:12.064 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:11:12.064 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:12.064 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:12.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:12.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:11:12.064 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:11:12.064 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:11:12.065 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:11:12.065 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:11:12.065 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:11:12.065 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:11:12.065 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:11:12.065 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:11:12.065 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:12.066 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:12.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:12.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:11:12.066 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:11:12.066 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:11:12.066 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:11:12.066 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:11:12.067 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:11:12.067 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:11:12.067 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:11:12.067 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:11:12.067 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:12.067 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:12.068 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:12.068 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:11:12.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:12.068 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:11:12.068 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:11:12.068 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:11:12.068 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:12.069 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:11:12.070 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945439.492888_50.1 target= 2020-11-09T18:11:17.071 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:11:17.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:11:17.072 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:11:17.118 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:11:17.119 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:11:17.371 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:11:17.372 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:11:17.372 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:11:17.372 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:11:17.372 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:11:17.372 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:11:17.373 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:17.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:17.373 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:11:17.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:11:17.373 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:11:17.373 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:11:17.373 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:11:17.374 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:11:17.374 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:11:17.374 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:11:17.374 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:11:17.374 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:11:17.374 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:17.375 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:17.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:17.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:11:17.375 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:11:17.375 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:11:17.375 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:11:17.375 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:11:17.376 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:11:17.376 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:11:17.376 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:11:17.376 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:11:17.376 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:17.376 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:17.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:17.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:11:17.377 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:11:17.377 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:11:17.377 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:11:17.377 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:11:17.377 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:11:17.378 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:11:17.378 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:11:17.378 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:11:17.378 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:17.378 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:17.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:17.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:11:17.379 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:11:17.379 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:11:17.379 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:11:17.379 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:11:17.379 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:11:17.379 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:11:17.379 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:11:17.380 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:11:17.380 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:17.380 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:17.380 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:17.380 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:11:17.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:17.381 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:11:17.381 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:11:17.381 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:11:17.381 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:17.381 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:11:17.382 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945439.492888_50.1 target= 2020-11-09T18:11:22.383 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:11:22.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:11:22.385 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:11:22.425 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:11:22.426 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:11:22.684 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:11:22.685 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:11:22.685 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:11:22.685 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:11:22.685 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:11:22.686 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:11:22.686 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:22.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:22.686 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:11:22.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:11:22.686 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:11:22.687 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:11:22.687 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:11:22.687 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:11:22.687 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:11:22.687 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:11:22.687 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:11:22.687 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:11:22.688 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:22.688 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:22.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:22.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:11:22.688 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:11:22.688 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:11:22.688 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:11:22.689 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:11:22.689 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:11:22.689 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:11:22.689 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:11:22.689 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:11:22.689 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:22.689 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:22.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:22.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:11:22.690 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:11:22.690 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:11:22.690 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:11:22.690 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:11:22.691 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:11:22.691 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:11:22.691 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:11:22.691 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:11:22.691 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:22.691 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:22.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:22.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:11:22.692 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:11:22.692 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:11:22.692 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:11:22.692 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:11:22.692 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:11:22.693 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:11:22.693 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:11:22.693 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:11:22.693 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:22.693 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:22.693 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:22.693 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:11:22.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:22.695 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:11:22.695 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:11:22.695 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:11:22.695 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:22.695 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:11:22.697 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945439.492888_50.1 target= 2020-11-09T18:11:27.698 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:11:27.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:11:27.699 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:11:27.744 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:11:27.745 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:11:28.003 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:11:28.004 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:11:28.004 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:11:28.004 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:11:28.005 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:11:28.005 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:11:28.005 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:28.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:28.006 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:11:28.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:11:28.006 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:11:28.007 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:11:28.007 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:11:28.007 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:11:28.008 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:11:28.008 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:11:28.008 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:11:28.008 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:11:28.009 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:28.009 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:28.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:28.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:11:28.010 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:11:28.010 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:11:28.010 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:11:28.011 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:11:28.011 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:11:28.011 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:11:28.012 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:11:28.012 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:11:28.012 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:28.013 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:28.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:28.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:11:28.013 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:11:28.014 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:11:28.014 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:11:28.014 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:11:28.015 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:11:28.015 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:11:28.015 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:11:28.015 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:11:28.015 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:28.015 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:28.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:28.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:11:28.016 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:11:28.016 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:11:28.016 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:11:28.016 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:11:28.016 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:11:28.017 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:11:28.017 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:11:28.017 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:11:28.017 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:28.017 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:28.017 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:28.017 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:11:28.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:28.018 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:11:28.018 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:11:28.018 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:11:28.018 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:28.019 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:11:28.020 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945439.492888_50.1 target= 2020-11-09T18:11:33.022 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:11:33.022 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:11:33.023 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:11:33.064 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:11:33.064 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:11:33.308 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:11:33.309 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:11:33.309 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:11:33.309 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:11:33.310 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:11:33.310 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:11:33.310 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:33.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:33.311 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:11:33.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:11:33.311 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:11:33.312 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:11:33.312 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:11:33.312 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:11:33.313 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:11:33.313 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:11:33.313 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:11:33.313 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:11:33.314 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:33.314 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:33.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:33.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:11:33.315 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:11:33.315 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:11:33.315 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:11:33.316 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:11:33.316 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:11:33.316 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:11:33.317 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:11:33.317 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:11:33.317 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:33.318 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:33.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:33.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:11:33.318 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:11:33.319 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:11:33.319 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:11:33.319 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:11:33.320 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:11:33.320 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:11:33.320 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:11:33.321 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:11:33.321 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:33.322 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:33.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:33.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:11:33.322 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:11:33.323 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:11:33.323 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:11:33.323 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:11:33.324 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:11:33.324 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:11:33.324 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:11:33.324 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:11:33.325 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:33.325 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:33.325 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:33.326 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:11:33.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:33.326 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:11:33.327 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:11:33.327 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:11:33.327 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:33.327 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:11:33.329 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945439.492888_50.1 target= 2020-11-09T18:11:38.330 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:11:38.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:11:38.331 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:11:38.371 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:11:38.372 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:11:38.622 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:11:38.622 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:11:38.623 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:11:38.623 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:11:38.623 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:11:38.623 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:11:38.623 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:38.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:38.623 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:11:38.624 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:11:38.624 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:11:38.624 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:11:38.624 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:11:38.624 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:11:38.624 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:11:38.624 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:11:38.625 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:11:38.625 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:11:38.625 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:38.625 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:38.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:38.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:11:38.625 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:11:38.626 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:11:38.626 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:11:38.626 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:11:38.626 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:11:38.626 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:11:38.626 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:11:38.626 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:11:38.626 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:38.627 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:38.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:38.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:11:38.627 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:11:38.627 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:11:38.627 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:11:38.627 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:11:38.628 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:11:38.628 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:11:38.628 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:11:38.628 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:11:38.628 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:38.628 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:38.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:38.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:11:38.629 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:11:38.629 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:11:38.629 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:11:38.629 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:11:38.629 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:11:38.629 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:11:38.629 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:11:38.630 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:11:38.630 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:38.630 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:38.630 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:38.630 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:11:38.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:38.630 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:11:38.631 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:11:38.631 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:11:38.631 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:38.631 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:11:38.632 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945439.492888_50.1 target= 2020-11-09T18:11:43.634 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:11:43.634 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:11:43.635 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:11:43.675 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:11:43.676 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:11:43.940 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:11:43.941 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:11:43.941 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:11:43.941 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:11:43.941 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:11:43.941 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:11:43.942 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:43.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:43.942 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:11:43.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:11:43.943 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:11:43.943 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:11:43.943 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:11:43.943 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:11:43.943 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:11:43.943 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:11:43.943 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:11:43.944 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:11:43.944 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:43.944 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:43.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:43.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:11:43.944 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:11:43.944 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:11:43.945 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:11:43.945 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:11:43.945 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:11:43.945 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:11:43.945 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:11:43.945 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:11:43.945 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:43.946 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:43.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:43.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:11:43.946 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:11:43.946 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:11:43.946 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:11:43.946 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:11:43.947 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:11:43.947 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:11:43.947 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:11:43.947 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:11:43.947 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:43.947 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:43.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:43.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:11:43.948 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:11:43.948 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:11:43.948 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:11:43.948 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:11:43.948 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:11:43.948 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:11:43.948 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:11:43.950 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:11:43.950 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:43.950 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:43.951 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:43.951 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:11:43.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:43.951 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:11:43.951 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:11:43.951 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:11:43.951 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:43.952 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:11:43.953 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945439.492888_50.1 target= 2020-11-09T18:11:48.953 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:11:48.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:11:48.954 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:11:48.996 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:11:48.996 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:11:49.255 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:11:49.255 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:11:49.256 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:11:49.256 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:11:49.256 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:11:49.257 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:11:49.257 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:49.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:49.257 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:11:49.258 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:11:49.258 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:11:49.258 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:11:49.258 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:11:49.259 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:11:49.259 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:11:49.259 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:11:49.260 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:11:49.260 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:11:49.260 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:49.261 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:49.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:49.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:11:49.261 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:11:49.262 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:11:49.262 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:11:49.263 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:11:49.263 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:11:49.263 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:11:49.263 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:11:49.264 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:11:49.264 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:49.264 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:49.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:49.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:11:49.265 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:11:49.265 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:11:49.266 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:11:49.266 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:11:49.266 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:11:49.266 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:11:49.267 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:11:49.267 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:11:49.267 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:49.267 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:49.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:49.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:11:49.268 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:11:49.268 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:11:49.268 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:11:49.268 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:11:49.268 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:11:49.268 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:11:49.269 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:11:49.269 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:11:49.269 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:49.269 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:49.269 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:49.269 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:11:49.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:49.269 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:11:49.270 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:11:49.270 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:11:49.270 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:49.270 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:11:49.270 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945439.492888_50.1 target= 2020-11-09T18:11:54.272 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:11:54.273 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:11:54.273 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:11:54.315 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:11:54.315 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:11:54.574 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:11:54.574 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:11:54.575 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:11:54.576 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:11:54.576 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:11:54.577 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:11:54.577 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:54.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:54.578 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:11:54.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:11:54.579 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:11:54.579 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:11:54.580 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:11:54.580 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:11:54.580 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:11:54.581 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:11:54.581 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:11:54.581 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:11:54.582 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:54.582 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:54.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:54.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:11:54.584 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:11:54.584 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:11:54.584 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:11:54.585 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:11:54.585 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:11:54.585 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:11:54.586 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:11:54.586 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:11:54.586 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:54.587 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:54.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:54.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:11:54.588 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:11:54.588 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:11:54.588 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:11:54.589 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:11:54.589 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:11:54.590 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:11:54.590 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:11:54.590 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:11:54.591 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:54.591 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:54.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:54.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:11:54.592 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:11:54.593 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:11:54.593 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:11:54.593 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:11:54.594 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:11:54.594 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:11:54.594 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:11:54.595 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:11:54.595 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:11:54.595 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:54.596 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:54.596 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:11:54.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:11:54.596 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:11:54.596 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:11:54.597 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:11:54.597 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:11:54.597 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:11:54.598 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945439.492888_50.1 target= 2020-11-09T18:11:59.599 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:11:59.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:11:59.601 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:11:59.637 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:11:59.638 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:11:59.877 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:12:00.370 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:12:00.371 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:12:00.371 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:12:00.371 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:12:00.371 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:12:00.371 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:00.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:00.372 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:12:00.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:00.372 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:12:00.372 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:00.372 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:00.372 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:00.372 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:00.372 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:12:00.373 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:00.373 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:00.373 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:00.373 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:00.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:00.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:00.373 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:12:00.374 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:00.374 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:00.374 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:00.374 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:00.374 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:00.374 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:00.374 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:00.375 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:00.375 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:00.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:00.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:00.375 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:12:00.375 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:00.375 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:00.376 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:00.376 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:00.376 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:00.376 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:00.376 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:00.376 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:00.376 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:00.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:00.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:00.377 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:12:00.377 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:00.377 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:00.377 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:00.377 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:00.377 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:00.378 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:00.378 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:00.378 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:00.378 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:00.378 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:00.378 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:12:00.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:00.379 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:12:00.379 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:12:00.379 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:12:00.379 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:00.379 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:12:00.380 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945439.492888_50.1 target= 2020-11-09T18:12:05.381 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:05.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:05.382 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:12:05.423 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:12:05.424 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:12:05.658 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:12:05.658 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:12:05.658 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:12:05.659 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:12:05.659 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:12:05.659 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:12:05.659 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:05.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:05.659 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:12:05.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:05.660 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:12:05.660 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:05.660 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:05.661 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:05.661 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:05.661 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:12:05.662 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:05.662 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:05.662 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:05.662 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:05.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:05.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:05.663 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:12:05.663 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:05.663 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:05.664 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:05.664 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:05.664 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:05.664 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:05.664 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:05.664 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:05.665 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:05.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:05.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:05.665 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:12:05.665 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:05.665 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:05.666 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:05.666 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:05.666 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:05.666 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:05.666 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:05.666 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:05.667 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:05.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:05.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:05.667 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:12:05.667 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:05.667 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:05.668 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:05.668 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:05.668 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:05.668 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:05.668 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:05.668 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:05.669 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:05.669 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:05.669 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:12:05.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:05.669 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:12:05.669 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:12:05.669 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:12:05.670 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:05.670 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:12:05.671 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945439.492888_50.1 target= 2020-11-09T18:12:10.672 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:10.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:10.673 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:12:10.712 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:12:10.712 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:12:10.965 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:12:10.966 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:12:10.966 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:12:10.966 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:12:10.966 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:12:10.967 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:12:10.967 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:10.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:10.967 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:12:10.967 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:10.967 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:12:10.968 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:10.968 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:10.972 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:10.972 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:10.972 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:12:10.972 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:10.973 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:10.973 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:10.973 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:10.973 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:10.973 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:10.973 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:12:10.973 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:10.974 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:10.974 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:10.974 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:10.974 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:10.974 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:10.974 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:10.974 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:10.975 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:10.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:10.975 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:10.975 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:12:10.975 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:10.975 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:10.975 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:10.976 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:10.976 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:10.976 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:10.976 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:10.976 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:10.976 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:10.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:10.976 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:10.977 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:12:10.977 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:10.977 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:10.977 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:10.977 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:10.977 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:10.977 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:10.978 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:10.978 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:10.978 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:10.978 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:10.978 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:12:10.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:10.978 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:12:10.979 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:12:10.979 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:12:10.979 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:10.979 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:12:10.980 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945439.492888_50.1 target= 2020-11-09T18:12:15.981 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:15.981 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:15.982 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:12:16.023 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:12:16.023 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:12:16.273 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:12:16.274 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:12:16.274 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:12:16.274 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:12:16.274 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:12:16.274 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:12:16.274 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:16.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:16.275 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:12:16.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:16.275 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:12:16.275 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:16.275 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:16.275 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:16.275 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:16.276 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:12:16.276 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:16.276 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:16.276 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:16.276 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:16.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:16.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:16.277 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:12:16.277 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:16.277 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:16.277 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:16.277 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:16.277 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:16.277 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:16.278 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:16.278 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:16.278 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:16.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:16.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:16.278 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:12:16.278 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:16.279 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:16.279 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:16.279 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:16.279 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:16.279 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:16.279 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:16.279 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:16.280 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:16.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:16.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:16.280 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:12:16.280 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:16.280 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:16.280 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:16.280 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:16.281 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:16.281 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:16.281 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:16.281 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:16.281 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:16.281 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:16.281 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:12:16.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:16.282 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:12:16.282 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:12:16.282 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:12:16.283 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:16.283 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:12:16.284 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945439.492888_50.1 target= 2020-11-09T18:12:21.285 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:21.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:21.286 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:12:21.328 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:12:21.328 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:12:21.569 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:12:21.570 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:12:21.572 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:12:21.573 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:12:21.576 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:12:21.579 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:12:21.581 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:21.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:21.583 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:12:21.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:21.585 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:12:21.586 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:21.586 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:21.587 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:21.587 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:21.588 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:12:21.588 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:21.589 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:21.589 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:21.589 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:21.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:21.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:21.590 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:12:21.590 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:21.590 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:21.591 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:21.591 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:21.591 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:21.591 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:21.592 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:21.592 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:21.592 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:21.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:21.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:21.593 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:12:21.593 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:21.594 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:21.594 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:21.594 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:21.594 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:21.595 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:21.595 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:21.595 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:21.595 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:21.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:21.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:21.597 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:12:21.598 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:21.598 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:21.598 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:21.599 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:21.599 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:21.599 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:21.600 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:21.600 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:21.600 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:21.600 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:21.601 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:12:21.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:21.610 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:12:21.610 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:12:21.610 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:12:21.610 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:21.611 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:12:21.612 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945439.492888_50.1 target= 2020-11-09T18:12:26.613 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:26.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:26.614 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:12:26.654 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:12:26.655 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:12:26.889 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:12:26.889 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:12:26.890 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:12:26.890 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:12:26.890 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:12:26.890 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:12:26.890 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:26.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:26.891 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:12:26.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:26.891 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:12:26.891 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:26.891 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:26.891 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:26.891 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:26.892 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:12:26.892 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:26.892 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:26.892 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:26.892 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:26.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:26.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:26.893 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:12:26.893 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:26.893 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:26.893 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:26.893 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:26.893 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:26.893 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:26.894 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:26.894 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:26.894 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:26.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:26.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:26.894 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:12:26.894 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:26.895 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:26.895 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:26.896 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:26.896 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:26.897 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:26.897 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:26.897 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:26.897 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:26.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:26.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:26.897 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:12:26.898 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:26.898 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:26.898 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:26.898 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:26.899 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:26.899 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:26.899 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:26.899 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:26.899 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:26.899 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:26.900 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:12:26.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:26.900 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:12:26.900 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:12:26.900 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:12:26.900 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:26.900 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:12:26.901 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945439.492888_50.1 target= 2020-11-09T18:12:31.902 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:31.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:31.904 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:12:31.949 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:12:31.949 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:12:32.185 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:12:32.185 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:12:32.185 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:12:32.185 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:12:32.185 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:12:32.186 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:12:32.186 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:32.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:32.186 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:12:32.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:32.186 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:12:32.186 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:32.187 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:32.187 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:32.187 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:32.187 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:12:32.187 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:32.187 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:32.187 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:32.188 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:32.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:32.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:32.188 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:12:32.188 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:32.188 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:32.188 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:32.189 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:32.189 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:32.189 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:32.189 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:32.189 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:32.189 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:32.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:32.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:32.190 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:12:32.190 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:32.190 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:32.191 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:32.191 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:32.191 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:32.191 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:32.191 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:32.191 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:32.191 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:32.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:32.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:32.192 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:12:32.192 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:32.192 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:32.192 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:32.192 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:32.192 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:32.193 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:32.193 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:32.193 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:32.193 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:32.193 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:32.193 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:12:32.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:32.194 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:12:32.194 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:12:32.194 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:12:32.194 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:32.194 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:12:32.195 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945439.492888_50.1 target= 2020-11-09T18:12:37.196 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:37.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:37.197 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:12:37.238 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:12:37.239 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:12:37.485 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:12:37.486 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:12:37.486 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:12:37.486 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:12:37.486 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:12:37.486 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:12:37.486 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:37.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:37.487 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:12:37.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:37.487 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:12:37.487 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:37.487 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:37.487 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945439.492888_50.1", 2020-11-09T18:12:37.487 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:37.488 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:12:37.488 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:37.488 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:10:39.492888Z", 2020-11-09T18:12:37.488 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:37.488 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:37.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:37.488 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:37.489 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:12:37.489 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:37.489 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:37.489 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:37.489 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:37.489 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:37.489 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:37.490 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:37.490 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:37.490 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:37.490 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:37.490 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:37.490 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:12:37.490 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:37.491 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:37.491 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:37.491 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:37.491 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:37.491 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:37.491 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:37.491 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:37.492 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:37.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:37.492 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:37.492 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:12:37.493 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:37.493 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:37.493 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:37.493 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:37.493 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:37.493 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:37.493 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:37.494 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:37.494 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:37.499 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:37.499 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:12:37.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:37.500 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:12:37.500 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:12:37.500 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:12:37.500 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:37.500 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:12:37.503 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-09T18:12:37.504 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-09T18:12:37.509 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:37.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:37.510 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:12:37.566 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:12:37.567 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:12:37.870 INFO:teuthology.orchestra.run.smithi051.stdout:{ 2020-11-09T18:12:37.871 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_status": { 2020-11-09T18:12:37.871 INFO:teuthology.orchestra.run.smithi051.stdout: "info": { 2020-11-09T18:12:37.871 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "sync", 2020-11-09T18:12:37.871 INFO:teuthology.orchestra.run.smithi051.stdout: "num_shards": 4, 2020-11-09T18:12:37.871 INFO:teuthology.orchestra.run.smithi051.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:12:37.872 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:12:37.872 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:12:37.872 INFO:teuthology.orchestra.run.smithi051.stdout: "markers": [ 2020-11-09T18:12:37.872 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:12:37.872 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 0, 2020-11-09T18:12:37.872 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:12:37.872 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:12:37.873 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945439.492888_50.1", 2020-11-09T18:12:37.873 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:12:37.873 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 1, 2020-11-09T18:12:37.873 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:12:37.873 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:10:39.492888Z", 2020-11-09T18:12:37.873 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:12:37.873 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:12:37.874 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:12:37.874 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:12:37.874 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 1, 2020-11-09T18:12:37.874 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:12:37.874 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:12:37.874 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:12:37.874 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:12:37.875 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:12:37.875 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:12:37.875 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:12:37.875 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:12:37.876 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:12:37.876 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:12:37.876 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:12:37.876 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 2, 2020-11-09T18:12:37.876 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:12:37.876 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:12:37.877 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:12:37.877 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:12:37.877 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:12:37.877 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:12:37.877 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:12:37.877 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:12:37.877 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:12:37.878 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:12:37.878 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:12:37.878 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 3, 2020-11-09T18:12:37.878 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:12:37.878 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:12:37.878 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:12:37.878 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:12:37.878 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:12:37.879 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:12:37.879 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:12:37.879 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:12:37.879 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:12:37.879 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:12:37.879 INFO:teuthology.orchestra.run.smithi051.stdout: ] 2020-11-09T18:12:37.879 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:12:37.880 INFO:teuthology.orchestra.run.smithi051.stdout: "full_sync": { 2020-11-09T18:12:37.880 INFO:teuthology.orchestra.run.smithi051.stdout: "total": 1, 2020-11-09T18:12:37.880 INFO:teuthology.orchestra.run.smithi051.stdout: "complete": 1 2020-11-09T18:12:37.880 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:12:37.880 INFO:teuthology.orchestra.run.smithi051.stdout:} 2020-11-09T18:12:37.881 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-09T18:12:37.881 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-09T18:12:37.882 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:37.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:37.883 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:12:37.923 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:12:37.923 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:12:38.192 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:12:38.193 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:12:38.193 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:12:38.194 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:12:38.194 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:12:38.194 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:12:38.194 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:38.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:38.195 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:12:38.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:38.196 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:12:38.196 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:38.196 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:38.196 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945439.492888_50.1", 2020-11-09T18:12:38.197 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:38.197 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:12:38.197 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:38.198 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:10:39.492888Z", 2020-11-09T18:12:38.203 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:38.203 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:38.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:38.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:38.203 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:12:38.204 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:38.204 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:38.204 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:38.204 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:38.204 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:38.204 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:38.204 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:38.204 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:38.205 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:38.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:38.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:38.205 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:12:38.205 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:38.205 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:38.205 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:38.206 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:38.206 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:38.206 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:38.206 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:38.206 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:38.206 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:38.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:38.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:38.207 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:12:38.207 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:38.207 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:38.207 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:38.207 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:38.207 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:38.208 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:38.208 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:38.208 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:38.208 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:38.208 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:38.208 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:12:38.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:38.208 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:12:38.209 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:12:38.209 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:12:38.209 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:38.209 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:12:38.210 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-09T18:12:38.210 INFO:rgw_multi.tests:meta checkpoint 2020-11-09T18:12:38.211 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:38.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:38.211 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:12:38.251 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:12:38.251 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:12:38.497 INFO:teuthology.orchestra.run.smithi051.stderr:No --period given, using current period=a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9 2020-11-09T18:12:38.524 INFO:teuthology.orchestra.run.smithi051.stdout:[ 2020-11-09T18:12:38.525 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:12:38.525 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945439.492888_50.1", 2020-11-09T18:12:38.525 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "2020-11-09T18:10:39.492888Z" 2020-11-09T18:12:38.526 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:12:38.526 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:12:38.526 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:12:38.526 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:12:38.526 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:12:38.526 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:12:38.527 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:12:38.527 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:12:38.527 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:12:38.527 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:12:38.527 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:12:38.527 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:12:38.528 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:12:38.528 INFO:teuthology.orchestra.run.smithi051.stdout:] 2020-11-09T18:12:38.528 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-09T18:12:38.528 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:38.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:38.529 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:12:38.570 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:12:38.571 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:12:38.834 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:12:38.835 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:12:38.835 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:12:38.835 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:12:38.835 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:12:38.835 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:12:38.836 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:38.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:38.836 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:12:38.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:38.836 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:12:38.836 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:38.836 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:38.837 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945439.492888_50.1", 2020-11-09T18:12:38.837 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:38.837 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:12:38.837 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:38.837 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:10:39.492888Z", 2020-11-09T18:12:38.837 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:38.837 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:38.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:38.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:38.838 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:12:38.838 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:38.838 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:38.852 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:38.853 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:38.853 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:38.854 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:38.854 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:38.854 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:38.854 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:38.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:38.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:38.855 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:12:38.855 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:38.856 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:38.856 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:38.856 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:38.856 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:38.857 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:38.857 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:38.857 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:38.857 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:38.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:38.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:38.867 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:12:38.868 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:38.868 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:38.868 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:38.869 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:38.869 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:38.869 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:38.870 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:38.870 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:38.870 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:38.870 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:38.871 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:12:38.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:38.871 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:12:38.871 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:12:38.872 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:12:38.872 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:38.872 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:12:38.873 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-09T18:12:38.873 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-09T18:12:38.874 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:38.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:38.875 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:12:38.930 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:12:38.931 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:12:39.182 INFO:teuthology.orchestra.run.smithi051.stdout:{ 2020-11-09T18:12:39.183 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_status": { 2020-11-09T18:12:39.183 INFO:teuthology.orchestra.run.smithi051.stdout: "info": { 2020-11-09T18:12:39.183 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "sync", 2020-11-09T18:12:39.184 INFO:teuthology.orchestra.run.smithi051.stdout: "num_shards": 4, 2020-11-09T18:12:39.184 INFO:teuthology.orchestra.run.smithi051.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:12:39.184 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:12:39.184 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:12:39.184 INFO:teuthology.orchestra.run.smithi051.stdout: "markers": [ 2020-11-09T18:12:39.184 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:12:39.184 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 0, 2020-11-09T18:12:39.185 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:12:39.185 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:12:39.185 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945439.492888_50.1", 2020-11-09T18:12:39.185 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:12:39.185 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 1, 2020-11-09T18:12:39.186 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:12:39.190 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:10:39.492888Z", 2020-11-09T18:12:39.190 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:12:39.191 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:12:39.191 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:12:39.191 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:12:39.206 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 1, 2020-11-09T18:12:39.206 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:12:39.206 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:12:39.206 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:12:39.207 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:12:39.207 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:12:39.207 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:12:39.207 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:12:39.207 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:12:39.207 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:12:39.207 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:12:39.208 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:12:39.208 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 2, 2020-11-09T18:12:39.208 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:12:39.208 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:12:39.208 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:12:39.208 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:12:39.208 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:12:39.209 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:12:39.209 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:12:39.209 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:12:39.209 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:12:39.209 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:12:39.209 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:12:39.209 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 3, 2020-11-09T18:12:39.210 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:12:39.214 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:12:39.214 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:12:39.215 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:12:39.215 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:12:39.215 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:12:39.215 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:12:39.215 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:12:39.215 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:12:39.215 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:12:39.216 INFO:teuthology.orchestra.run.smithi051.stdout: ] 2020-11-09T18:12:39.216 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:12:39.216 INFO:teuthology.orchestra.run.smithi051.stdout: "full_sync": { 2020-11-09T18:12:39.216 INFO:teuthology.orchestra.run.smithi051.stdout: "total": 1, 2020-11-09T18:12:39.216 INFO:teuthology.orchestra.run.smithi051.stdout: "complete": 1 2020-11-09T18:12:39.216 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:12:39.216 INFO:teuthology.orchestra.run.smithi051.stdout:} 2020-11-09T18:12:39.217 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-09T18:12:39.226 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-09T18:12:39.226 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:39.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:39.227 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:12:39.278 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:12:39.279 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:12:39.538 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:12:39.538 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:12:39.539 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:12:39.539 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:12:39.539 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:12:39.539 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:12:39.539 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:39.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:39.539 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:12:39.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:39.540 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:12:39.540 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:39.540 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:39.540 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945439.492888_50.1", 2020-11-09T18:12:39.540 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:39.541 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:12:39.541 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:39.541 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:10:39.492888Z", 2020-11-09T18:12:39.541 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:39.541 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:39.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:39.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:39.542 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:12:39.542 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:39.543 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:39.543 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:39.543 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:39.543 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:39.543 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:39.543 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:39.544 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:39.544 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:39.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:39.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:39.544 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:12:39.544 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:39.544 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:39.545 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:39.545 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:39.545 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:39.545 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:39.546 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:39.546 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:39.546 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:39.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:39.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:39.547 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:12:39.547 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:39.548 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:39.548 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:39.548 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:39.548 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:39.548 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:39.549 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:39.549 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:39.549 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:39.549 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:39.549 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:12:39.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:39.549 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:12:39.550 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:12:39.552 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:12:39.552 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:39.552 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:12:39.554 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-09T18:12:39.555 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on 'tasks.rgw_multi.tests' with args=[] 2020-11-09T18:12:39.561 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ymzeui-1 2020-11-09T18:12:43.578 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ymzeui-2 2020-11-09T18:12:47.594 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ymzeui-3 2020-11-09T18:12:47.658 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:47.659 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:47.660 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:12:47.696 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:12:47.696 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:12:47.911 INFO:teuthology.orchestra.run.smithi051.stderr:No --period given, using current period=a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9 2020-11-09T18:12:47.945 INFO:teuthology.orchestra.run.smithi051.stdout:[ 2020-11-09T18:12:47.945 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:12:47.946 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945439.492888_50.1", 2020-11-09T18:12:47.947 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "2020-11-09T18:10:39.492888Z" 2020-11-09T18:12:47.947 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:12:47.948 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:12:47.948 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945567.636634_61.1", 2020-11-09T18:12:47.949 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "2020-11-09T18:12:47.636634Z" 2020-11-09T18:12:47.949 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:12:47.950 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:12:47.950 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:12:47.950 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:12:47.950 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:12:47.951 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:12:47.951 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:12:47.951 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:12:47.952 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:12:47.952 INFO:teuthology.orchestra.run.smithi051.stdout:] 2020-11-09T18:12:47.954 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-09T18:12:47.955 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:47.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:47.956 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:12:48.002 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:12:48.003 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:12:48.246 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:12:48.247 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:12:48.247 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:12:48.247 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:12:48.247 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:12:48.247 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:12:48.248 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:48.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:48.248 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:12:48.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:48.248 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:12:48.248 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:48.248 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:48.249 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945439.492888_50.1", 2020-11-09T18:12:48.249 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:48.249 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:12:48.249 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:48.249 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:10:39.492888Z", 2020-11-09T18:12:48.249 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:48.250 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:48.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:48.250 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:48.250 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:12:48.250 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:48.251 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:48.251 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945567.636634_61.1", 2020-11-09T18:12:48.251 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:48.251 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:48.251 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:48.251 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:47.636634Z", 2020-11-09T18:12:48.251 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:48.252 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:48.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:48.252 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:48.252 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:12:48.252 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:48.252 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:48.252 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:48.253 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:48.253 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:48.253 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:48.253 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:48.253 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:48.253 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:48.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:48.254 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:48.254 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:12:48.254 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:48.254 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:48.254 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:48.254 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:48.254 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:48.255 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:48.255 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:48.255 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:48.255 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:48.255 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:48.255 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:12:48.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:48.255 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:12:48.256 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:12:48.256 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:12:48.256 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:48.256 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:12:48.257 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-09T18:12:48.258 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-09T18:12:48.258 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:48.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:48.259 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:12:48.302 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:12:48.302 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:12:48.551 INFO:teuthology.orchestra.run.smithi051.stdout:{ 2020-11-09T18:12:48.552 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_status": { 2020-11-09T18:12:48.552 INFO:teuthology.orchestra.run.smithi051.stdout: "info": { 2020-11-09T18:12:48.552 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "sync", 2020-11-09T18:12:48.552 INFO:teuthology.orchestra.run.smithi051.stdout: "num_shards": 4, 2020-11-09T18:12:48.552 INFO:teuthology.orchestra.run.smithi051.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:12:48.552 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:12:48.552 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:12:48.553 INFO:teuthology.orchestra.run.smithi051.stdout: "markers": [ 2020-11-09T18:12:48.553 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:12:48.553 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 0, 2020-11-09T18:12:48.553 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:12:48.553 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:12:48.553 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945439.492888_50.1", 2020-11-09T18:12:48.553 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:12:48.554 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 1, 2020-11-09T18:12:48.554 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:12:48.554 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:10:39.492888Z", 2020-11-09T18:12:48.554 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:12:48.554 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:12:48.554 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:12:48.554 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:12:48.555 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 1, 2020-11-09T18:12:48.555 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:12:48.555 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:12:48.555 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945567.636634_61.1", 2020-11-09T18:12:48.555 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:12:48.555 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:12:48.555 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:12:48.556 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:47.636634Z", 2020-11-09T18:12:48.556 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:12:48.556 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:12:48.556 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:12:48.556 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:12:48.556 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 2, 2020-11-09T18:12:48.556 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:12:48.556 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:12:48.557 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:12:48.557 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:12:48.557 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:12:48.557 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:12:48.557 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:12:48.557 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:12:48.557 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:12:48.558 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:12:48.558 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:12:48.558 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 3, 2020-11-09T18:12:48.558 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:12:48.558 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:12:48.558 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:12:48.558 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:12:48.559 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:12:48.559 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:12:48.559 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:12:48.559 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:12:48.559 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:12:48.559 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:12:48.559 INFO:teuthology.orchestra.run.smithi051.stdout: ] 2020-11-09T18:12:48.560 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:12:48.560 INFO:teuthology.orchestra.run.smithi051.stdout: "full_sync": { 2020-11-09T18:12:48.560 INFO:teuthology.orchestra.run.smithi051.stdout: "total": 1, 2020-11-09T18:12:48.560 INFO:teuthology.orchestra.run.smithi051.stdout: "complete": 1 2020-11-09T18:12:48.560 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:12:48.560 INFO:teuthology.orchestra.run.smithi051.stdout:} 2020-11-09T18:12:48.561 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-09T18:12:48.562 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-09T18:12:48.562 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:48.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:48.563 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:12:48.601 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:12:48.601 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:12:48.866 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:12:48.866 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:12:48.866 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:12:48.867 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:12:48.867 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:12:48.867 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:12:48.867 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:48.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:48.867 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:12:48.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:48.868 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:12:48.868 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:48.868 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:48.868 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945439.492888_50.1", 2020-11-09T18:12:48.868 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:48.869 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:12:48.869 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:48.869 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:10:39.492888Z", 2020-11-09T18:12:48.869 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:48.869 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:48.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:48.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:48.870 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:12:48.870 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:48.870 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:48.870 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945567.636634_61.1", 2020-11-09T18:12:48.870 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:48.871 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:48.871 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:48.871 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:47.636634Z", 2020-11-09T18:12:48.871 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:48.871 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:48.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:48.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:48.872 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:12:48.873 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:48.873 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:48.873 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:48.873 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:48.873 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:48.873 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:48.874 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:48.874 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:48.874 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:48.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:48.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:48.874 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:12:48.875 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:48.875 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:48.875 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:48.875 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:48.875 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:48.875 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:48.876 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:48.876 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:48.877 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:48.877 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:48.877 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:12:48.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:48.878 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:12:48.878 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:12:48.878 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:12:48.878 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:48.878 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:12:48.879 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-09T18:12:50.351 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_create ... ok 2020-11-09T18:12:50.354 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ymzeui-4 2020-11-09T18:12:50.382 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ymzeui-5 2020-11-09T18:12:50.434 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ymzeui-6 2020-11-09T18:12:50.519 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:50.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:50.520 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:12:50.557 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:12:50.557 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:12:50.782 INFO:teuthology.orchestra.run.smithi051.stderr:No --period given, using current period=a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9 2020-11-09T18:12:50.813 INFO:teuthology.orchestra.run.smithi051.stdout:[ 2020-11-09T18:12:50.813 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:12:50.813 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945439.492888_50.1", 2020-11-09T18:12:50.813 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "2020-11-09T18:10:39.492888Z" 2020-11-09T18:12:50.813 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:12:50.814 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:12:50.814 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945570.489122_74.1", 2020-11-09T18:12:50.814 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "2020-11-09T18:12:50.489122Z" 2020-11-09T18:12:50.814 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:12:50.814 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:12:50.814 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:12:50.814 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:12:50.815 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:12:50.815 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:12:50.815 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:12:50.815 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:12:50.815 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:12:50.815 INFO:teuthology.orchestra.run.smithi051.stdout:] 2020-11-09T18:12:50.816 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-09T18:12:50.817 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:50.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:50.818 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:12:50.858 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:12:50.859 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:12:51.105 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:12:51.105 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:12:51.105 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:12:51.105 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:12:51.106 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:12:51.106 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:12:51.106 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:51.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:51.106 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:12:51.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:51.106 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:12:51.107 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:51.107 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:51.107 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945439.492888_50.1", 2020-11-09T18:12:51.107 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:51.107 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:12:51.107 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:51.107 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:10:39.492888Z", 2020-11-09T18:12:51.108 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:51.108 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:51.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:51.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:51.108 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:12:51.108 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:51.108 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:51.109 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945570.489122_74.1", 2020-11-09T18:12:51.109 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:51.109 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:51.109 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:51.109 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:50.489122Z", 2020-11-09T18:12:51.109 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:51.109 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:51.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:51.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:51.110 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:12:51.110 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:51.110 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:51.110 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:51.110 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:51.110 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:51.111 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:51.111 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:51.111 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:51.111 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:51.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:51.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:51.112 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:12:51.112 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:51.112 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:51.112 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:51.113 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:51.113 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:51.113 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:51.113 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:51.113 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:51.113 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:51.113 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:51.114 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:12:51.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:51.114 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:12:51.114 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:12:51.114 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:12:51.114 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:51.114 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:12:51.115 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-09T18:12:51.116 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-09T18:12:51.116 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:51.117 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:51.117 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:12:51.165 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:12:51.165 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:12:51.432 INFO:teuthology.orchestra.run.smithi051.stdout:{ 2020-11-09T18:12:51.432 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_status": { 2020-11-09T18:12:51.433 INFO:teuthology.orchestra.run.smithi051.stdout: "info": { 2020-11-09T18:12:51.433 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "sync", 2020-11-09T18:12:51.433 INFO:teuthology.orchestra.run.smithi051.stdout: "num_shards": 4, 2020-11-09T18:12:51.433 INFO:teuthology.orchestra.run.smithi051.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:12:51.433 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:12:51.433 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:12:51.433 INFO:teuthology.orchestra.run.smithi051.stdout: "markers": [ 2020-11-09T18:12:51.434 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:12:51.434 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 0, 2020-11-09T18:12:51.434 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:12:51.434 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:12:51.434 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945439.492888_50.1", 2020-11-09T18:12:51.434 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:12:51.435 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 1, 2020-11-09T18:12:51.435 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:12:51.435 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:10:39.492888Z", 2020-11-09T18:12:51.435 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:12:51.435 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:12:51.437 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:12:51.437 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:12:51.437 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 1, 2020-11-09T18:12:51.437 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:12:51.437 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:12:51.437 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945570.489122_74.1", 2020-11-09T18:12:51.437 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:12:51.438 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:12:51.438 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:12:51.438 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:50.489122Z", 2020-11-09T18:12:51.438 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:12:51.438 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:12:51.438 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:12:51.439 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:12:51.439 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 2, 2020-11-09T18:12:51.439 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:12:51.439 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:12:51.439 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:12:51.439 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:12:51.439 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:12:51.439 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:12:51.440 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:12:51.440 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:12:51.440 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:12:51.440 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:12:51.440 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:12:51.440 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 3, 2020-11-09T18:12:51.440 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:12:51.441 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:12:51.441 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:12:51.441 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:12:51.441 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:12:51.441 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:12:51.441 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:12:51.441 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:12:51.441 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:12:51.442 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:12:51.442 INFO:teuthology.orchestra.run.smithi051.stdout: ] 2020-11-09T18:12:51.443 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:12:51.443 INFO:teuthology.orchestra.run.smithi051.stdout: "full_sync": { 2020-11-09T18:12:51.443 INFO:teuthology.orchestra.run.smithi051.stdout: "total": 1, 2020-11-09T18:12:51.443 INFO:teuthology.orchestra.run.smithi051.stdout: "complete": 1 2020-11-09T18:12:51.443 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:12:51.443 INFO:teuthology.orchestra.run.smithi051.stdout:} 2020-11-09T18:12:51.444 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-09T18:12:51.449 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-09T18:12:51.449 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:51.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:51.450 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:12:51.493 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:12:51.493 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:12:51.765 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:12:51.765 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:12:51.766 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:12:51.766 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:12:51.766 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:12:51.766 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:12:51.766 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:51.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:51.767 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:12:51.767 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:51.767 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:12:51.767 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:51.767 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:51.767 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945439.492888_50.1", 2020-11-09T18:12:51.768 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:51.768 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:12:51.768 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:51.768 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:10:39.492888Z", 2020-11-09T18:12:51.768 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:51.768 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:51.768 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:51.769 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:51.769 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:12:51.769 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:51.769 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:51.769 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945570.489122_74.1", 2020-11-09T18:12:51.769 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:51.769 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:51.769 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:51.771 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:50.489122Z", 2020-11-09T18:12:51.771 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:51.771 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:51.772 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:51.772 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:51.772 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:12:51.772 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:51.772 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:51.772 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:51.772 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:51.773 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:51.773 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:51.773 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:51.773 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:51.773 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:51.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:51.773 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:51.774 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:12:51.774 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:51.774 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:51.774 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:51.775 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:51.775 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:51.775 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:51.775 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:51.775 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:51.775 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:51.775 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:51.776 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:12:51.776 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:51.776 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:12:51.776 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:12:51.776 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:12:51.776 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:51.777 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:12:51.778 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-09T18:12:56.600 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:56.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:56.601 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:12:56.639 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:12:56.640 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:12:56.865 INFO:teuthology.orchestra.run.smithi051.stderr:No --period given, using current period=a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9 2020-11-09T18:12:56.901 INFO:teuthology.orchestra.run.smithi051.stdout:[ 2020-11-09T18:12:56.902 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:12:56.902 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945439.492888_50.1", 2020-11-09T18:12:56.902 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "2020-11-09T18:10:39.492888Z" 2020-11-09T18:12:56.902 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:12:56.903 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:12:56.903 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945570.489122_74.1", 2020-11-09T18:12:56.903 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "2020-11-09T18:12:50.489122Z" 2020-11-09T18:12:56.904 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:12:56.904 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:12:56.904 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:12:56.905 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:12:56.905 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:12:56.905 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:12:56.905 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:12:56.906 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:12:56.906 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:12:56.906 INFO:teuthology.orchestra.run.smithi051.stdout:] 2020-11-09T18:12:56.908 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-09T18:12:56.908 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:56.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:56.910 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:12:56.960 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:12:56.961 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:12:57.214 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:12:57.214 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:12:57.214 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:12:57.214 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:12:57.214 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:12:57.215 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:12:57.215 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:57.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:57.215 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:12:57.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:57.215 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:12:57.216 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:57.216 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:57.216 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945439.492888_50.1", 2020-11-09T18:12:57.216 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:57.217 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:12:57.217 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:57.217 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:10:39.492888Z", 2020-11-09T18:12:57.217 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:57.217 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:57.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:57.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:57.218 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:12:57.218 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:57.218 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:57.218 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945570.489122_74.1", 2020-11-09T18:12:57.219 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:57.219 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:57.219 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:57.219 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:50.489122Z", 2020-11-09T18:12:57.219 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:57.220 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:57.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:57.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:57.221 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:12:57.221 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:57.221 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:57.221 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:57.221 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:57.221 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:57.221 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:57.222 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:57.222 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:57.222 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:57.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:57.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:57.222 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:12:57.223 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:57.223 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:57.223 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:57.223 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:57.223 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:57.223 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:57.223 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:57.224 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:57.224 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:57.224 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:57.224 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:12:57.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:57.224 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:12:57.225 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:12:57.225 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:12:57.225 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:57.225 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:12:57.226 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-09T18:12:57.226 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-09T18:12:57.227 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:57.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:57.228 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:12:57.271 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:12:57.271 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:12:57.547 INFO:teuthology.orchestra.run.smithi051.stdout:{ 2020-11-09T18:12:57.548 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_status": { 2020-11-09T18:12:57.548 INFO:teuthology.orchestra.run.smithi051.stdout: "info": { 2020-11-09T18:12:57.548 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "sync", 2020-11-09T18:12:57.548 INFO:teuthology.orchestra.run.smithi051.stdout: "num_shards": 4, 2020-11-09T18:12:57.548 INFO:teuthology.orchestra.run.smithi051.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:12:57.548 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:12:57.548 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:12:57.549 INFO:teuthology.orchestra.run.smithi051.stdout: "markers": [ 2020-11-09T18:12:57.549 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:12:57.549 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 0, 2020-11-09T18:12:57.549 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:12:57.549 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:12:57.549 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945439.492888_50.1", 2020-11-09T18:12:57.549 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:12:57.550 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 1, 2020-11-09T18:12:57.550 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:12:57.550 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:10:39.492888Z", 2020-11-09T18:12:57.550 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:12:57.550 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:12:57.550 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:12:57.550 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:12:57.551 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 1, 2020-11-09T18:12:57.551 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:12:57.551 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:12:57.551 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945570.489122_74.1", 2020-11-09T18:12:57.551 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:12:57.552 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:12:57.553 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:12:57.553 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:50.489122Z", 2020-11-09T18:12:57.553 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:12:57.553 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:12:57.554 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:12:57.554 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:12:57.554 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 2, 2020-11-09T18:12:57.555 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:12:57.555 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:12:57.555 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:12:57.556 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:12:57.556 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:12:57.556 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:12:57.556 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:12:57.557 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:12:57.557 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:12:57.557 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:12:57.558 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:12:57.558 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 3, 2020-11-09T18:12:57.558 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:12:57.559 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:12:57.559 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:12:57.559 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:12:57.560 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:12:57.560 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:12:57.560 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:12:57.561 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:12:57.561 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:12:57.561 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:12:57.561 INFO:teuthology.orchestra.run.smithi051.stdout: ] 2020-11-09T18:12:57.562 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:12:57.563 INFO:teuthology.orchestra.run.smithi051.stdout: "full_sync": { 2020-11-09T18:12:57.563 INFO:teuthology.orchestra.run.smithi051.stdout: "total": 1, 2020-11-09T18:12:57.563 INFO:teuthology.orchestra.run.smithi051.stdout: "complete": 1 2020-11-09T18:12:57.564 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:12:57.564 INFO:teuthology.orchestra.run.smithi051.stdout:} 2020-11-09T18:12:57.566 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-09T18:12:57.567 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-09T18:12:57.567 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:57.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:57.569 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:12:57.610 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:12:57.611 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:12:57.879 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:12:57.879 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:12:57.879 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:12:57.879 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:12:57.880 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:12:57.880 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:12:57.880 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:57.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:57.880 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:12:57.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:57.880 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:12:57.881 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:57.881 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:57.881 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945439.492888_50.1", 2020-11-09T18:12:57.881 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:57.881 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:12:57.881 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:57.881 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:10:39.492888Z", 2020-11-09T18:12:57.882 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:57.882 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:57.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:57.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:57.882 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:12:57.882 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:57.882 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:57.883 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945570.489122_74.1", 2020-11-09T18:12:57.883 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:57.883 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:57.883 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:57.883 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:50.489122Z", 2020-11-09T18:12:57.883 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:57.884 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:57.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:57.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:57.884 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:12:57.884 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:57.884 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:57.884 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:57.885 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:57.885 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:57.885 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:57.885 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:57.885 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:57.885 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:57.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:57.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:12:57.886 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:12:57.886 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:12:57.886 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:12:57.886 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:12:57.886 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:12:57.886 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:12:57.887 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:12:57.887 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:12:57.887 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:12:57.887 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:57.887 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:57.887 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:12:57.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:12:57.888 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:12:57.888 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:12:57.888 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:12:57.888 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:12:57.888 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:12:57.889 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-09T18:12:59.289 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_recreate ... ok 2020-11-09T18:12:59.290 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ymzeui-7 2020-11-09T18:12:59.370 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ymzeui-8 2020-11-09T18:12:59.425 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ymzeui-9 2020-11-09T18:12:59.489 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:59.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:59.490 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:12:59.528 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:12:59.528 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:12:59.729 INFO:teuthology.orchestra.run.smithi051.stderr:No --period given, using current period=a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9 2020-11-09T18:12:59.758 INFO:teuthology.orchestra.run.smithi051.stdout:[ 2020-11-09T18:12:59.758 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:12:59.758 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945439.492888_50.1", 2020-11-09T18:12:59.759 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "2020-11-09T18:10:39.492888Z" 2020-11-09T18:12:59.759 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:12:59.759 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:12:59.759 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945579.467305_88.1", 2020-11-09T18:12:59.759 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "2020-11-09T18:12:59.467305Z" 2020-11-09T18:12:59.760 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:12:59.760 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:12:59.760 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:12:59.760 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:12:59.760 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:12:59.760 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:12:59.761 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:12:59.761 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:12:59.761 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:12:59.761 INFO:teuthology.orchestra.run.smithi051.stdout:] 2020-11-09T18:12:59.763 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-09T18:12:59.763 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:59.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:12:59.764 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:12:59.803 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:12:59.803 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:13:00.055 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:13:00.056 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:13:00.056 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:13:00.056 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:13:00.056 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:13:00.056 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:13:00.056 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:00.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:00.057 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:13:00.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:00.057 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:13:00.057 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:00.057 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:00.057 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945439.492888_50.1", 2020-11-09T18:13:00.058 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:00.058 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:13:00.058 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:00.058 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:10:39.492888Z", 2020-11-09T18:13:00.058 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:00.058 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:00.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:00.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:00.059 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:13:00.059 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:00.059 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:00.059 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945579.467305_88.1", 2020-11-09T18:13:00.059 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:00.060 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:00.060 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:00.060 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:59.467305Z", 2020-11-09T18:13:00.060 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:00.060 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:00.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:00.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:00.061 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:13:00.061 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:00.061 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:00.061 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:13:00.061 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:00.061 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:00.061 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:00.062 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:13:00.062 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:00.062 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:00.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:00.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:00.062 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:13:00.063 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:00.063 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:00.063 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:13:00.063 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:00.063 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:00.063 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:00.063 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:13:00.064 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:00.064 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:00.064 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:00.064 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:13:00.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:00.064 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:13:00.064 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:13:00.065 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:13:00.065 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:00.065 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:13:00.066 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-09T18:13:00.067 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-09T18:13:00.067 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:00.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:00.068 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:13:00.106 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:13:00.107 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:13:00.373 INFO:teuthology.orchestra.run.smithi051.stdout:{ 2020-11-09T18:13:00.374 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_status": { 2020-11-09T18:13:00.374 INFO:teuthology.orchestra.run.smithi051.stdout: "info": { 2020-11-09T18:13:00.374 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "sync", 2020-11-09T18:13:00.374 INFO:teuthology.orchestra.run.smithi051.stdout: "num_shards": 4, 2020-11-09T18:13:00.374 INFO:teuthology.orchestra.run.smithi051.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:13:00.375 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:13:00.375 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:00.375 INFO:teuthology.orchestra.run.smithi051.stdout: "markers": [ 2020-11-09T18:13:00.375 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:00.375 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 0, 2020-11-09T18:13:00.375 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:13:00.375 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:13:00.376 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945439.492888_50.1", 2020-11-09T18:13:00.376 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:13:00.376 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 1, 2020-11-09T18:13:00.376 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:13:00.376 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:10:39.492888Z", 2020-11-09T18:13:00.376 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:13:00.376 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:00.377 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:00.377 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:00.377 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 1, 2020-11-09T18:13:00.377 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:13:00.377 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:13:00.377 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945579.467305_88.1", 2020-11-09T18:13:00.377 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:13:00.377 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:13:00.378 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:13:00.378 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:59.467305Z", 2020-11-09T18:13:00.378 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:13:00.378 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:00.378 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:00.378 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:00.378 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 2, 2020-11-09T18:13:00.379 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:13:00.379 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:13:00.379 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:13:00.379 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:13:00.379 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:13:00.379 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:13:00.379 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:13:00.380 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:13:00.380 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:00.380 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:00.380 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:00.380 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 3, 2020-11-09T18:13:00.380 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:13:00.380 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:13:00.380 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:13:00.381 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:13:00.381 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:13:00.381 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:13:00.381 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:13:00.381 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:13:00.381 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:00.381 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:00.382 INFO:teuthology.orchestra.run.smithi051.stdout: ] 2020-11-09T18:13:00.382 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:00.382 INFO:teuthology.orchestra.run.smithi051.stdout: "full_sync": { 2020-11-09T18:13:00.382 INFO:teuthology.orchestra.run.smithi051.stdout: "total": 1, 2020-11-09T18:13:00.383 INFO:teuthology.orchestra.run.smithi051.stdout: "complete": 1 2020-11-09T18:13:00.383 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:00.383 INFO:teuthology.orchestra.run.smithi051.stdout:} 2020-11-09T18:13:00.384 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-09T18:13:00.384 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-09T18:13:00.385 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:00.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:00.386 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:13:00.426 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:13:00.426 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:13:00.698 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:13:00.698 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:13:00.698 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:13:00.698 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:13:00.699 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:13:00.699 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:13:00.699 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:00.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:00.699 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:13:00.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:00.699 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:13:00.700 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:00.700 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:00.700 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945439.492888_50.1", 2020-11-09T18:13:00.700 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:00.700 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:13:00.700 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:00.700 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:10:39.492888Z", 2020-11-09T18:13:00.701 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:00.701 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:00.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:00.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:00.701 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:13:00.701 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:00.701 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:00.701 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945579.467305_88.1", 2020-11-09T18:13:00.702 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:00.702 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:00.702 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:00.702 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:59.467305Z", 2020-11-09T18:13:00.702 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:00.702 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:00.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:00.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:00.703 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:13:00.703 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:00.703 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:00.703 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:13:00.703 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:00.703 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:00.704 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:00.704 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:13:00.704 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:00.704 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:00.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:00.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:00.705 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:13:00.705 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:00.705 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:00.705 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:13:00.705 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:00.706 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:00.706 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:00.706 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:13:00.706 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:00.706 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:00.706 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:00.706 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:13:00.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:00.707 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:13:00.707 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:13:00.707 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:13:00.707 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:00.707 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:13:00.708 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-09T18:13:02.472 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:02.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:02.474 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:13:02.528 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:13:02.529 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:13:02.753 INFO:teuthology.orchestra.run.smithi051.stderr:No --period given, using current period=a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9 2020-11-09T18:13:02.785 INFO:teuthology.orchestra.run.smithi051.stdout:[ 2020-11-09T18:13:02.786 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:02.786 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945439.492888_50.1", 2020-11-09T18:13:02.787 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "2020-11-09T18:10:39.492888Z" 2020-11-09T18:13:02.787 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:02.788 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:02.788 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:13:02.788 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "2020-11-09T18:13:02.432939Z" 2020-11-09T18:13:02.789 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:02.789 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:02.789 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:13:02.790 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:13:02.790 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:02.791 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:02.791 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:13:02.791 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:13:02.791 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:02.791 INFO:teuthology.orchestra.run.smithi051.stdout:] 2020-11-09T18:13:02.792 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-09T18:13:02.793 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:02.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:02.794 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:13:02.848 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:13:02.848 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:13:03.107 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:13:03.108 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:13:03.108 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:13:03.108 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:13:03.108 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:13:03.108 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:13:03.109 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:03.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:03.109 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:13:03.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:03.109 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:13:03.109 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:03.109 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:03.110 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945439.492888_50.1", 2020-11-09T18:13:03.110 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:03.110 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:13:03.110 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:03.110 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:10:39.492888Z", 2020-11-09T18:13:03.110 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:03.110 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:03.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:03.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:03.111 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:13:03.111 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:03.111 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:03.111 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:13:03.111 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:03.111 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:03.112 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:03.112 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:13:03.112 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:03.112 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:03.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:03.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:03.112 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:13:03.113 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:03.113 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:03.113 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:13:03.113 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:03.113 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:03.113 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:03.113 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:13:03.114 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:03.114 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:03.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:03.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:03.115 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:13:03.115 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:03.115 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:03.116 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:13:03.116 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:03.116 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:03.117 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:03.117 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:13:03.117 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:03.118 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:03.118 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:03.118 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:13:03.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:03.119 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:13:03.119 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:13:03.120 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:13:03.120 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:03.120 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:13:03.122 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-09T18:13:03.123 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-09T18:13:03.123 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:03.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:03.125 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:13:03.172 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:13:03.172 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:13:03.429 INFO:teuthology.orchestra.run.smithi051.stdout:{ 2020-11-09T18:13:03.429 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_status": { 2020-11-09T18:13:03.429 INFO:teuthology.orchestra.run.smithi051.stdout: "info": { 2020-11-09T18:13:03.429 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "sync", 2020-11-09T18:13:03.429 INFO:teuthology.orchestra.run.smithi051.stdout: "num_shards": 4, 2020-11-09T18:13:03.430 INFO:teuthology.orchestra.run.smithi051.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:13:03.430 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:13:03.430 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:03.430 INFO:teuthology.orchestra.run.smithi051.stdout: "markers": [ 2020-11-09T18:13:03.430 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:03.430 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 0, 2020-11-09T18:13:03.432 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:13:03.432 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:13:03.433 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945439.492888_50.1", 2020-11-09T18:13:03.433 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:13:03.433 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 1, 2020-11-09T18:13:03.433 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:13:03.433 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:10:39.492888Z", 2020-11-09T18:13:03.433 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:13:03.434 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:03.434 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:03.434 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:03.434 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 1, 2020-11-09T18:13:03.434 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:13:03.434 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:13:03.434 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:13:03.435 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:13:03.435 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:13:03.435 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:13:03.435 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:13:03.435 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:13:03.436 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:03.436 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:03.436 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:03.436 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 2, 2020-11-09T18:13:03.436 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:13:03.436 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:13:03.436 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:13:03.437 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:13:03.437 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:13:03.437 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:13:03.437 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:13:03.437 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:13:03.437 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:03.437 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:03.438 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:03.439 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 3, 2020-11-09T18:13:03.439 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:13:03.439 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:13:03.440 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:13:03.440 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:13:03.440 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:13:03.440 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:13:03.441 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:13:03.441 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:13:03.441 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:03.441 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:03.442 INFO:teuthology.orchestra.run.smithi051.stdout: ] 2020-11-09T18:13:03.443 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:03.443 INFO:teuthology.orchestra.run.smithi051.stdout: "full_sync": { 2020-11-09T18:13:03.443 INFO:teuthology.orchestra.run.smithi051.stdout: "total": 1, 2020-11-09T18:13:03.444 INFO:teuthology.orchestra.run.smithi051.stdout: "complete": 1 2020-11-09T18:13:03.445 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:03.445 INFO:teuthology.orchestra.run.smithi051.stdout:} 2020-11-09T18:13:03.447 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-09T18:13:03.448 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-09T18:13:03.449 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:03.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:03.450 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:13:03.493 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:13:03.494 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:13:03.760 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:13:03.760 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:13:03.760 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:13:03.760 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:13:03.760 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:13:03.761 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:13:03.761 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:03.761 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:03.761 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:13:03.761 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:03.761 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:13:03.762 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:03.762 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:03.762 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945439.492888_50.1", 2020-11-09T18:13:03.762 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:03.762 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:13:03.763 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:03.763 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:10:39.492888Z", 2020-11-09T18:13:03.763 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:03.763 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:03.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:03.763 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:03.764 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:13:03.764 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:03.764 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:03.764 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:13:03.764 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:03.764 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:03.765 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:03.765 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:13:03.765 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:03.765 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:03.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:03.765 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:03.766 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:13:03.766 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:03.766 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:03.766 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:13:03.766 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:03.766 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:03.767 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:03.767 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:13:03.767 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:03.767 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:03.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:03.767 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:03.767 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:13:03.768 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:03.768 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:03.769 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:13:03.769 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:03.769 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:03.769 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:03.769 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:13:03.770 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:03.770 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:03.770 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:03.770 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:13:03.770 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:03.770 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:13:03.770 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:13:03.771 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:13:03.771 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:03.771 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:13:03.772 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-09T18:13:05.851 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_remove ... ok 2020-11-09T18:13:05.852 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ymzeui-10 2020-11-09T18:13:06.267 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ymzeui-11 2020-11-09T18:13:07.239 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ymzeui-12 2020-11-09T18:13:08.308 INFO:tasks.rgw.c1.client.0.smithi051.stdout:Parse error at line 1: 2020-11-09T18:13:08.308 INFO:tasks.rgw.c1.client.0.smithi051.stdout:syntax error 2020-11-09T18:13:08.356 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_object_sync ... ERROR 2020-11-09T18:13:08.357 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ymzeui-13 2020-11-09T18:13:08.400 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ymzeui-14 2020-11-09T18:13:09.412 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ymzeui-15 2020-11-09T18:13:09.936 INFO:tasks.rgw.c1.client.0.smithi051.stdout:Parse error at line 1: 2020-11-09T18:13:09.936 INFO:tasks.rgw.c1.client.0.smithi051.stdout:syntax error 2020-11-09T18:13:09.984 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_object_delete ... ERROR 2020-11-09T18:13:09.985 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ymzeui-16 2020-11-09T18:13:10.027 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ymzeui-17 2020-11-09T18:13:10.096 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ymzeui-18 2020-11-09T18:13:10.206 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:10.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:10.208 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:13:10.249 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:13:10.249 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:13:10.483 INFO:teuthology.orchestra.run.smithi051.stderr:No --period given, using current period=a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9 2020-11-09T18:13:10.519 INFO:teuthology.orchestra.run.smithi051.stdout:[ 2020-11-09T18:13:10.520 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:10.520 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945590.199376_99.1", 2020-11-09T18:13:10.520 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "2020-11-09T18:13:10.199376Z" 2020-11-09T18:13:10.520 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:10.520 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:10.520 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:13:10.521 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "2020-11-09T18:13:02.432939Z" 2020-11-09T18:13:10.521 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:10.521 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:10.521 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:13:10.521 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:13:10.521 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:10.521 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:10.522 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:13:10.522 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:13:10.522 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:10.522 INFO:teuthology.orchestra.run.smithi051.stdout:] 2020-11-09T18:13:10.523 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-09T18:13:10.523 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:10.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:10.524 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:13:10.566 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:13:10.567 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:13:10.808 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:13:10.808 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:13:10.809 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:13:10.809 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:13:10.809 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:13:10.809 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:13:10.810 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:10.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:10.810 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:13:10.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:10.810 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:13:10.811 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:10.811 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:10.811 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945590.199376_99.1", 2020-11-09T18:13:10.811 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:10.811 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:13:10.812 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:10.812 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:10.199376Z", 2020-11-09T18:13:10.812 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:10.812 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:10.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:10.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:10.813 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:13:10.813 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:10.813 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:10.814 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:13:10.814 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:10.814 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:10.814 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:10.814 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:13:10.814 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:10.814 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:10.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:10.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:10.815 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:13:10.815 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:10.815 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:10.815 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:13:10.815 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:10.816 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:10.816 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:10.816 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:13:10.816 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:10.816 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:10.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:10.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:10.817 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:13:10.817 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:10.817 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:10.817 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:13:10.817 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:10.817 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:10.818 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:10.818 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:13:10.818 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:10.818 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:10.818 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:10.819 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:13:10.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:10.819 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:13:10.819 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:13:10.819 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:13:10.820 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:10.820 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:13:10.821 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-09T18:13:10.822 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-09T18:13:10.823 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:10.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:10.824 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:13:10.864 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:13:10.864 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:13:11.131 INFO:teuthology.orchestra.run.smithi051.stdout:{ 2020-11-09T18:13:11.131 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_status": { 2020-11-09T18:13:11.131 INFO:teuthology.orchestra.run.smithi051.stdout: "info": { 2020-11-09T18:13:11.131 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "sync", 2020-11-09T18:13:11.132 INFO:teuthology.orchestra.run.smithi051.stdout: "num_shards": 4, 2020-11-09T18:13:11.132 INFO:teuthology.orchestra.run.smithi051.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:13:11.132 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:13:11.132 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:11.132 INFO:teuthology.orchestra.run.smithi051.stdout: "markers": [ 2020-11-09T18:13:11.132 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:11.133 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 0, 2020-11-09T18:13:11.133 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:13:11.133 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:13:11.133 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945590.199376_99.1", 2020-11-09T18:13:11.133 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:13:11.133 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 1, 2020-11-09T18:13:11.134 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:13:11.134 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:10.199376Z", 2020-11-09T18:13:11.134 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:13:11.134 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:11.134 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:11.134 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:11.135 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 1, 2020-11-09T18:13:11.135 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:13:11.135 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:13:11.135 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:13:11.135 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:13:11.135 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:13:11.135 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:13:11.136 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:13:11.136 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:13:11.136 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:11.136 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:11.136 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:11.136 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 2, 2020-11-09T18:13:11.137 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:13:11.137 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:13:11.137 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:13:11.137 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:13:11.137 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:13:11.137 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:13:11.137 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:13:11.138 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:13:11.138 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:11.138 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:11.138 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:11.138 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 3, 2020-11-09T18:13:11.139 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:13:11.139 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:13:11.139 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:13:11.139 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:13:11.139 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:13:11.139 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:13:11.139 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:13:11.140 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:13:11.140 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:11.140 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:11.140 INFO:teuthology.orchestra.run.smithi051.stdout: ] 2020-11-09T18:13:11.140 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:11.140 INFO:teuthology.orchestra.run.smithi051.stdout: "full_sync": { 2020-11-09T18:13:11.141 INFO:teuthology.orchestra.run.smithi051.stdout: "total": 1, 2020-11-09T18:13:11.141 INFO:teuthology.orchestra.run.smithi051.stdout: "complete": 1 2020-11-09T18:13:11.141 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:11.141 INFO:teuthology.orchestra.run.smithi051.stdout:} 2020-11-09T18:13:11.142 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-09T18:13:11.143 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-09T18:13:11.143 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:11.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:11.144 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:13:11.184 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:13:11.185 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:13:11.448 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:13:11.449 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:13:11.449 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:13:11.449 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:13:11.449 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:13:11.449 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:13:11.450 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:11.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:11.450 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:13:11.450 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:11.450 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:13:11.450 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:11.450 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:11.451 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945590.199376_99.1", 2020-11-09T18:13:11.451 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:11.451 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:13:11.451 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:11.451 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:10.199376Z", 2020-11-09T18:13:11.451 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:11.451 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:11.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:11.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:11.452 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:13:11.452 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:11.452 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:11.452 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:13:11.452 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:11.452 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:11.453 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:11.453 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:13:11.453 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:11.453 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:11.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:11.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:11.453 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:13:11.454 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:11.454 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:11.454 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:13:11.454 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:11.454 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:11.454 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:11.454 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:13:11.455 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:11.455 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:11.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:11.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:11.455 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:13:11.455 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:11.455 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:11.455 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:13:11.456 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:11.456 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:11.456 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:11.456 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:13:11.456 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:11.456 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:11.456 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:11.457 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:13:11.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:11.457 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:13:11.457 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:13:11.457 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:13:11.457 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:11.457 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:13:11.458 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-09T18:13:12.243 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_versioned_object_incremental_sync ... ERROR 2020-11-09T18:13:12.297 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:12.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:12.298 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:13:12.337 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:13:12.337 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:13:12.549 INFO:teuthology.orchestra.run.smithi051.stderr:No --period given, using current period=a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9 2020-11-09T18:13:12.577 INFO:teuthology.orchestra.run.smithi051.stdout:[ 2020-11-09T18:13:12.577 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:12.578 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945592.300193_106.1", 2020-11-09T18:13:12.578 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "2020-11-09T18:13:12.300193Z" 2020-11-09T18:13:12.578 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:12.578 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:12.578 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:13:12.578 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "2020-11-09T18:13:02.432939Z" 2020-11-09T18:13:12.578 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:12.579 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:12.579 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:13:12.579 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:13:12.579 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:12.579 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:12.579 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:13:12.579 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:13:12.580 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:12.580 INFO:teuthology.orchestra.run.smithi051.stdout:] 2020-11-09T18:13:12.581 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-09T18:13:12.581 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:12.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:12.582 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:13:12.622 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:13:12.622 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:13:12.871 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:13:12.871 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:13:12.871 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:13:12.872 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:13:12.872 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:13:12.872 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:13:12.872 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:12.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:12.872 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:13:12.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:12.873 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:13:12.873 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:12.873 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:12.873 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945592.300193_106.1", 2020-11-09T18:13:12.873 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:12.873 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:13:12.873 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:12.874 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:12.300193Z", 2020-11-09T18:13:12.874 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:12.874 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:12.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:12.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:12.874 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:13:12.875 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:12.875 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:12.875 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:13:12.875 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:12.875 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:12.875 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:12.876 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:13:12.876 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:12.876 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:12.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:12.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:12.876 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:13:12.876 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:12.877 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:12.877 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:13:12.877 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:12.877 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:12.877 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:12.877 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:13:12.877 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:12.878 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:12.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:12.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:12.878 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:13:12.878 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:12.879 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:12.879 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:13:12.879 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:12.879 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:12.879 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:12.879 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:13:12.879 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:12.880 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:12.880 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:12.880 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:13:12.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:12.880 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:13:12.880 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:13:12.880 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:13:12.881 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:12.881 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:13:12.882 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-09T18:13:12.882 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-09T18:13:12.882 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:12.883 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:12.883 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:13:12.923 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:13:12.924 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:13:13.196 INFO:teuthology.orchestra.run.smithi051.stdout:{ 2020-11-09T18:13:13.197 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_status": { 2020-11-09T18:13:13.197 INFO:teuthology.orchestra.run.smithi051.stdout: "info": { 2020-11-09T18:13:13.197 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "sync", 2020-11-09T18:13:13.197 INFO:teuthology.orchestra.run.smithi051.stdout: "num_shards": 4, 2020-11-09T18:13:13.198 INFO:teuthology.orchestra.run.smithi051.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:13:13.198 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:13:13.198 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:13.198 INFO:teuthology.orchestra.run.smithi051.stdout: "markers": [ 2020-11-09T18:13:13.198 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:13.198 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 0, 2020-11-09T18:13:13.199 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:13:13.199 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:13:13.199 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945592.300193_106.1", 2020-11-09T18:13:13.199 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:13:13.199 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 1, 2020-11-09T18:13:13.199 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:13:13.199 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:12.300193Z", 2020-11-09T18:13:13.200 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:13:13.200 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:13.200 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:13.200 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:13.200 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 1, 2020-11-09T18:13:13.200 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:13:13.200 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:13:13.201 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:13:13.201 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:13:13.201 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:13:13.201 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:13:13.201 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:13:13.201 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:13:13.201 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:13.201 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:13.202 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:13.202 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 2, 2020-11-09T18:13:13.202 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:13:13.202 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:13:13.202 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:13:13.203 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:13:13.203 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:13:13.203 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:13:13.203 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:13:13.203 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:13:13.203 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:13.203 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:13.204 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:13.204 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 3, 2020-11-09T18:13:13.204 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:13:13.204 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:13:13.204 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:13:13.204 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:13:13.204 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:13:13.205 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:13:13.205 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:13:13.205 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:13:13.205 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:13.205 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:13.205 INFO:teuthology.orchestra.run.smithi051.stdout: ] 2020-11-09T18:13:13.205 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:13.205 INFO:teuthology.orchestra.run.smithi051.stdout: "full_sync": { 2020-11-09T18:13:13.206 INFO:teuthology.orchestra.run.smithi051.stdout: "total": 1, 2020-11-09T18:13:13.206 INFO:teuthology.orchestra.run.smithi051.stdout: "complete": 1 2020-11-09T18:13:13.206 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:13.206 INFO:teuthology.orchestra.run.smithi051.stdout:} 2020-11-09T18:13:13.207 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-09T18:13:13.208 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-09T18:13:13.209 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:13.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:13.210 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:13:13.259 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:13:13.260 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:13:13.609 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:13:13.609 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:13:13.610 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:13:13.610 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:13:13.610 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:13:13.610 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:13:13.610 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:13.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:13.610 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:13:13.611 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:13.611 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:13:13.611 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:13.611 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:13.611 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945592.300193_106.1", 2020-11-09T18:13:13.611 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:13.611 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:13:13.612 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:13.612 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:12.300193Z", 2020-11-09T18:13:13.612 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:13.612 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:13.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:13.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:13.612 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:13:13.613 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:13.613 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:13.613 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:13:13.613 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:13.613 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:13.613 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:13.613 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:13:13.613 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:13.614 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:13.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:13.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:13.614 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:13:13.614 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:13.615 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:13.616 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:13:13.616 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:13.616 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:13.616 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:13.616 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:13:13.616 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:13.616 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:13.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:13.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:13.617 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:13:13.617 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:13.617 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:13.617 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:13:13.617 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:13.618 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:13.618 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:13.618 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:13:13.618 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:13.618 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:13.618 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:13.618 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:13:13.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:13.619 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:13:13.619 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:13:13.619 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:13:13.619 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:13.619 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:13:13.620 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-09T18:13:14.015 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ERROR 2020-11-09T18:13:14.049 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:14.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:14.050 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:13:14.090 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:13:14.090 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:13:14.301 INFO:teuthology.orchestra.run.smithi051.stderr:No --period given, using current period=a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9 2020-11-09T18:13:14.336 INFO:teuthology.orchestra.run.smithi051.stdout:[ 2020-11-09T18:13:14.336 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:14.336 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945594.050612_111.1", 2020-11-09T18:13:14.336 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "2020-11-09T18:13:14.050612Z" 2020-11-09T18:13:14.337 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:14.337 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:14.337 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:13:14.337 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "2020-11-09T18:13:02.432939Z" 2020-11-09T18:13:14.337 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:14.337 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:14.337 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:13:14.338 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:13:14.338 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:14.338 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:14.338 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:13:14.338 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:13:14.338 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:14.338 INFO:teuthology.orchestra.run.smithi051.stdout:] 2020-11-09T18:13:14.339 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-09T18:13:14.340 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:14.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:14.341 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:13:14.383 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:13:14.383 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:13:14.625 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:13:14.626 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:13:14.626 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:13:14.626 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:13:14.626 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:13:14.627 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:13:14.627 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:14.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:14.627 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:13:14.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:14.627 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:13:14.628 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:14.628 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:14.628 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945594.050612_111.1", 2020-11-09T18:13:14.628 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:14.628 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:13:14.628 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:14.628 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:14.050612Z", 2020-11-09T18:13:14.628 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:14.629 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:14.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:14.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:14.629 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:13:14.629 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:14.629 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:14.629 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:13:14.630 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:14.630 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:14.630 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:14.630 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:13:14.630 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:14.630 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:14.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:14.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:14.631 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:13:14.631 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:14.631 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:14.631 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:13:14.631 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:14.631 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:14.631 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:14.632 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:13:14.632 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:14.632 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:14.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:14.632 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:14.632 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:13:14.632 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:14.633 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:14.633 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:13:14.633 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:14.633 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:14.633 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:14.633 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:13:14.633 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:14.634 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:14.634 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:14.634 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:13:14.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:14.634 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:13:14.634 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:13:14.634 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:13:14.634 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:14.635 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:13:14.636 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-09T18:13:14.636 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-09T18:13:14.636 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:14.637 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:14.637 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:13:14.679 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:13:14.680 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:13:14.948 INFO:teuthology.orchestra.run.smithi051.stdout:{ 2020-11-09T18:13:14.949 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_status": { 2020-11-09T18:13:14.949 INFO:teuthology.orchestra.run.smithi051.stdout: "info": { 2020-11-09T18:13:14.949 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "sync", 2020-11-09T18:13:14.949 INFO:teuthology.orchestra.run.smithi051.stdout: "num_shards": 4, 2020-11-09T18:13:14.949 INFO:teuthology.orchestra.run.smithi051.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:13:14.949 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:13:14.950 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:14.950 INFO:teuthology.orchestra.run.smithi051.stdout: "markers": [ 2020-11-09T18:13:14.950 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:14.950 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 0, 2020-11-09T18:13:14.950 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:13:14.950 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:13:14.950 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945594.050612_111.1", 2020-11-09T18:13:14.951 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:13:14.951 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 1, 2020-11-09T18:13:14.951 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:13:14.951 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:14.050612Z", 2020-11-09T18:13:14.951 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:13:14.951 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:14.951 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:14.952 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:14.952 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 1, 2020-11-09T18:13:14.952 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:13:14.952 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:13:14.952 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:13:14.952 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:13:14.952 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:13:14.953 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:13:14.953 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:13:14.953 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:13:14.953 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:14.953 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:14.953 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:14.953 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 2, 2020-11-09T18:13:14.954 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:13:14.954 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:13:14.954 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:13:14.954 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:13:14.954 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:13:14.954 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:13:14.954 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:13:14.955 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:13:14.955 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:14.955 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:14.955 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:14.955 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 3, 2020-11-09T18:13:14.955 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:13:14.955 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:13:14.956 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:13:14.956 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:13:14.956 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:13:14.956 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:13:14.956 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:13:14.956 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:13:14.956 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:14.957 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:14.957 INFO:teuthology.orchestra.run.smithi051.stdout: ] 2020-11-09T18:13:14.957 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:14.957 INFO:teuthology.orchestra.run.smithi051.stdout: "full_sync": { 2020-11-09T18:13:14.957 INFO:teuthology.orchestra.run.smithi051.stdout: "total": 1, 2020-11-09T18:13:14.957 INFO:teuthology.orchestra.run.smithi051.stdout: "complete": 1 2020-11-09T18:13:14.958 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:14.958 INFO:teuthology.orchestra.run.smithi051.stdout:} 2020-11-09T18:13:14.959 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-09T18:13:14.959 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-09T18:13:14.960 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:14.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:14.961 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:13:15.003 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:13:15.003 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:13:15.272 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:13:15.272 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:13:15.272 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:13:15.273 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:13:15.273 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:13:15.273 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:13:15.273 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:15.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:15.273 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:13:15.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:15.274 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:13:15.274 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:15.274 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:15.274 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945594.050612_111.1", 2020-11-09T18:13:15.274 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:15.274 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:13:15.275 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:15.275 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:14.050612Z", 2020-11-09T18:13:15.275 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:15.275 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:15.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:15.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:15.275 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:13:15.276 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:15.276 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:15.276 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:13:15.276 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:15.276 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:15.276 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:15.276 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:13:15.276 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:15.277 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:15.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:15.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:15.277 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:13:15.277 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:15.277 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:15.277 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:13:15.278 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:15.278 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:15.278 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:15.278 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:13:15.278 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:15.279 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:15.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:15.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:15.279 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:13:15.279 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:15.279 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:15.279 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:13:15.280 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:15.280 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:15.280 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:15.280 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:13:15.280 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:15.280 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:15.280 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:15.280 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:13:15.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:15.281 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:13:15.281 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:13:15.281 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:13:15.281 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:15.281 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:13:15.282 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-09T18:13:15.431 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_version_suspended_incremental_sync ... ERROR 2020-11-09T18:13:15.432 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ymzeui-21 2020-11-09T18:13:15.469 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ymzeui-22 2020-11-09T18:13:15.607 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ymzeui-23 2020-11-09T18:13:15.729 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:15.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:15.730 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:13:15.774 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:13:15.774 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:13:16.004 INFO:teuthology.orchestra.run.smithi051.stderr:No --period given, using current period=a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9 2020-11-09T18:13:16.038 INFO:teuthology.orchestra.run.smithi051.stdout:[ 2020-11-09T18:13:16.038 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:16.039 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945595.729371_130.1", 2020-11-09T18:13:16.039 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "2020-11-09T18:13:15.729371Z" 2020-11-09T18:13:16.039 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:16.039 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:16.039 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:13:16.039 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "2020-11-09T18:13:02.432939Z" 2020-11-09T18:13:16.040 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:16.040 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:16.040 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:13:16.040 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:13:16.040 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:16.040 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:16.040 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:13:16.041 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:13:16.041 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:16.041 INFO:teuthology.orchestra.run.smithi051.stdout:] 2020-11-09T18:13:16.042 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-09T18:13:16.042 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:16.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:16.043 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:13:16.083 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:13:16.084 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:13:16.316 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:13:16.317 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:13:16.317 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:13:16.317 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:13:16.317 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:13:16.317 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:13:16.317 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:16.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:16.318 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:13:16.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:16.318 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:13:16.318 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:16.318 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:16.319 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945595.729371_130.1", 2020-11-09T18:13:16.319 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:16.319 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:13:16.319 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:16.319 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:15.729371Z", 2020-11-09T18:13:16.319 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:16.320 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:16.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:16.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:16.320 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:13:16.320 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:16.320 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:16.320 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:13:16.321 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:16.321 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:16.321 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:16.321 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:13:16.321 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:16.321 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:16.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:16.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:16.322 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:13:16.322 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:16.322 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:16.322 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:13:16.322 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:16.323 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:16.323 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:16.323 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:13:16.323 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:16.323 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:16.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:16.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:16.324 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:13:16.324 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:16.324 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:16.324 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:13:16.324 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:16.324 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:16.325 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:16.325 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:13:16.325 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:16.325 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:16.325 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:16.325 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:13:16.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:16.326 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:13:16.326 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:13:16.326 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:13:16.326 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:16.326 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:13:16.327 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-09T18:13:16.328 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-09T18:13:16.328 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:16.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:16.329 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:13:16.371 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:13:16.372 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:13:16.627 INFO:teuthology.orchestra.run.smithi051.stdout:{ 2020-11-09T18:13:16.628 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_status": { 2020-11-09T18:13:16.628 INFO:teuthology.orchestra.run.smithi051.stdout: "info": { 2020-11-09T18:13:16.628 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "sync", 2020-11-09T18:13:16.628 INFO:teuthology.orchestra.run.smithi051.stdout: "num_shards": 4, 2020-11-09T18:13:16.628 INFO:teuthology.orchestra.run.smithi051.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:13:16.629 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:13:16.629 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:16.629 INFO:teuthology.orchestra.run.smithi051.stdout: "markers": [ 2020-11-09T18:13:16.629 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:16.629 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 0, 2020-11-09T18:13:16.629 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:13:16.629 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:13:16.629 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945595.729371_130.1", 2020-11-09T18:13:16.630 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:13:16.630 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 1, 2020-11-09T18:13:16.630 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:13:16.630 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:15.729371Z", 2020-11-09T18:13:16.630 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:13:16.630 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:16.630 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:16.631 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:16.631 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 1, 2020-11-09T18:13:16.631 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:13:16.631 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:13:16.631 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:13:16.631 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:13:16.631 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:13:16.632 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:13:16.632 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:13:16.632 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:13:16.632 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:16.632 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:16.632 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:16.632 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 2, 2020-11-09T18:13:16.633 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:13:16.633 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:13:16.633 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:13:16.633 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:13:16.634 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:13:16.634 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:13:16.634 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:13:16.634 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:13:16.634 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:16.634 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:16.635 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:16.635 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 3, 2020-11-09T18:13:16.635 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:13:16.635 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:13:16.635 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:13:16.635 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:13:16.635 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:13:16.636 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:13:16.636 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:13:16.636 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:13:16.636 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:16.636 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:16.636 INFO:teuthology.orchestra.run.smithi051.stdout: ] 2020-11-09T18:13:16.636 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:16.637 INFO:teuthology.orchestra.run.smithi051.stdout: "full_sync": { 2020-11-09T18:13:16.637 INFO:teuthology.orchestra.run.smithi051.stdout: "total": 1, 2020-11-09T18:13:16.637 INFO:teuthology.orchestra.run.smithi051.stdout: "complete": 1 2020-11-09T18:13:16.637 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:16.637 INFO:teuthology.orchestra.run.smithi051.stdout:} 2020-11-09T18:13:16.638 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-09T18:13:16.639 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-09T18:13:16.639 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:16.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:16.640 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:13:16.680 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:13:16.681 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:13:16.946 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:13:16.947 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:13:16.947 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:13:16.947 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:13:16.947 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:13:16.947 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:13:16.947 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:16.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:16.948 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:13:16.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:16.948 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:13:16.948 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:16.948 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:16.948 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945595.729371_130.1", 2020-11-09T18:13:16.949 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:16.949 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:13:16.949 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:16.949 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:15.729371Z", 2020-11-09T18:13:16.949 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:16.949 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:16.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:16.950 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:16.950 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:13:16.950 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:16.950 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:16.950 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:13:16.950 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:16.950 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:16.951 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:16.951 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:13:16.951 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:16.951 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:16.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:16.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:16.951 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:13:16.952 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:16.952 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:16.952 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:13:16.952 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:16.952 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:16.952 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:16.952 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:13:16.953 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:16.953 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:16.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:16.953 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:16.953 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:13:16.953 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:16.953 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:16.953 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:13:16.954 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:16.954 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:16.954 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:16.954 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:13:16.954 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:16.954 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:16.955 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:16.956 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:13:16.956 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:16.956 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:13:16.956 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:13:16.956 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:13:16.956 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:16.957 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:13:16.958 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-09T18:13:17.210 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_delete_marker_full_sync ... ERROR 2020-11-09T18:13:17.211 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ymzeui-24 2020-11-09T18:13:17.258 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ymzeui-25 2020-11-09T18:13:18.049 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ymzeui-26 2020-11-09T18:13:18.550 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:18.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:18.551 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:13:18.588 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:13:18.589 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:13:18.846 INFO:teuthology.orchestra.run.smithi051.stderr:No --period given, using current period=a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9 2020-11-09T18:13:18.880 INFO:teuthology.orchestra.run.smithi051.stdout:[ 2020-11-09T18:13:18.880 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:18.880 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945598.548940_155.1", 2020-11-09T18:13:18.881 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "2020-11-09T18:13:18.548940Z" 2020-11-09T18:13:18.881 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:18.881 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:18.881 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:13:18.881 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "2020-11-09T18:13:02.432939Z" 2020-11-09T18:13:18.881 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:18.882 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:18.882 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:13:18.882 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:13:18.882 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:18.882 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:18.883 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:13:18.883 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:13:18.883 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:18.883 INFO:teuthology.orchestra.run.smithi051.stdout:] 2020-11-09T18:13:18.885 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-09T18:13:18.885 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:18.886 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:18.886 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:13:18.927 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:13:18.927 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:13:19.193 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:13:19.193 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:13:19.194 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:13:19.194 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:13:19.194 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:13:19.194 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:13:19.194 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:19.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:19.195 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:13:19.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:19.195 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:13:19.195 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:19.195 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:19.195 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945598.548940_155.1", 2020-11-09T18:13:19.195 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:19.196 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:13:19.196 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:19.196 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:18.548940Z", 2020-11-09T18:13:19.196 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:19.196 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:19.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:19.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:19.196 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:13:19.197 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:19.197 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:19.197 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:13:19.197 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:19.197 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:19.198 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:19.198 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:13:19.198 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:19.198 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:19.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:19.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:19.199 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:13:19.199 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:19.199 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:19.199 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:13:19.199 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:19.199 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:19.200 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:19.200 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:13:19.200 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:19.200 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:19.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:19.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:19.200 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:13:19.201 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:19.201 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:19.201 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:13:19.201 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:19.201 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:19.201 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:19.201 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:13:19.202 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:19.202 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:19.202 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:19.202 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:13:19.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:19.202 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:13:19.202 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:13:19.202 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:13:19.203 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:19.203 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:13:19.204 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-09T18:13:19.204 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-09T18:13:19.205 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:19.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:19.205 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:13:19.246 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:13:19.247 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:13:19.511 INFO:teuthology.orchestra.run.smithi051.stdout:{ 2020-11-09T18:13:19.511 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_status": { 2020-11-09T18:13:19.512 INFO:teuthology.orchestra.run.smithi051.stdout: "info": { 2020-11-09T18:13:19.512 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "sync", 2020-11-09T18:13:19.512 INFO:teuthology.orchestra.run.smithi051.stdout: "num_shards": 4, 2020-11-09T18:13:19.512 INFO:teuthology.orchestra.run.smithi051.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:13:19.512 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:13:19.512 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:19.513 INFO:teuthology.orchestra.run.smithi051.stdout: "markers": [ 2020-11-09T18:13:19.513 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:19.513 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 0, 2020-11-09T18:13:19.513 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:13:19.513 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:13:19.513 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945598.548940_155.1", 2020-11-09T18:13:19.513 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:13:19.514 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 1, 2020-11-09T18:13:19.520 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:13:19.520 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:18.548940Z", 2020-11-09T18:13:19.520 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:13:19.520 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:19.520 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:19.521 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:19.521 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 1, 2020-11-09T18:13:19.521 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:13:19.521 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:13:19.521 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:13:19.521 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:13:19.521 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:13:19.522 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:13:19.522 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:13:19.522 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:13:19.522 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:19.522 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:19.522 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:19.522 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 2, 2020-11-09T18:13:19.523 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:13:19.523 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:13:19.523 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:13:19.523 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:13:19.523 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:13:19.523 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:13:19.523 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:13:19.524 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:13:19.524 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:19.524 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:19.524 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:19.524 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 3, 2020-11-09T18:13:19.524 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:13:19.524 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:13:19.525 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:13:19.525 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:13:19.525 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:13:19.525 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:13:19.525 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:13:19.525 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:13:19.525 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:19.525 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:19.526 INFO:teuthology.orchestra.run.smithi051.stdout: ] 2020-11-09T18:13:19.526 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:19.527 INFO:teuthology.orchestra.run.smithi051.stdout: "full_sync": { 2020-11-09T18:13:19.527 INFO:teuthology.orchestra.run.smithi051.stdout: "total": 1, 2020-11-09T18:13:19.527 INFO:teuthology.orchestra.run.smithi051.stdout: "complete": 1 2020-11-09T18:13:19.527 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:19.527 INFO:teuthology.orchestra.run.smithi051.stdout:} 2020-11-09T18:13:19.528 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-09T18:13:19.529 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-09T18:13:19.529 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:19.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:19.530 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:13:19.575 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:13:19.575 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:13:19.826 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:13:19.826 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:13:19.826 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:13:19.826 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:13:19.826 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:13:19.827 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:13:19.827 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:19.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:19.827 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:13:19.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:19.827 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:13:19.827 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:19.828 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:19.828 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945598.548940_155.1", 2020-11-09T18:13:19.828 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:19.828 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:13:19.828 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:19.828 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:18.548940Z", 2020-11-09T18:13:19.828 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:19.829 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:19.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:19.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:19.829 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:13:19.829 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:19.829 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:19.829 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:13:19.829 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:19.830 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:19.830 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:19.830 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:13:19.830 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:19.830 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:19.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:19.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:19.831 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:13:19.831 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:19.831 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:19.831 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:13:19.831 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:19.831 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:19.831 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:19.832 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:13:19.832 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:19.832 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:19.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:19.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:19.832 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:13:19.832 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:19.832 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:19.833 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:13:19.834 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:19.834 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:19.834 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:19.834 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:13:19.834 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:19.835 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:19.835 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:19.835 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:13:19.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:19.835 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:13:19.835 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:13:19.835 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:13:19.836 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:19.836 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:13:19.837 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-09T18:13:20.475 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_suspended_delete_marker_full_sync ... ERROR 2020-11-09T18:13:20.475 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ymzeui-27 2020-11-09T18:13:20.513 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ymzeui-28 2020-11-09T18:13:22.074 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ymzeui-29 2020-11-09T18:13:24.793 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_versioning ... ok 2020-11-09T18:13:24.794 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ymzeui-30 2020-11-09T18:13:25.418 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ymzeui-31 2020-11-09T18:13:25.843 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ymzeui-32 2020-11-09T18:13:26.546 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_acl ... ok 2020-11-09T18:13:26.547 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ymzeui-33 2020-11-09T18:13:26.578 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ymzeui-34 2020-11-09T18:13:26.631 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ymzeui-35 2020-11-09T18:13:29.338 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_cors ... ok 2020-11-09T18:13:29.339 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ymzeui-36 2020-11-09T18:13:29.678 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ymzeui-37 2020-11-09T18:13:30.108 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ymzeui-38 2020-11-09T18:13:30.163 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:30.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:30.165 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:13:30.201 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:13:30.201 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:13:30.422 INFO:teuthology.orchestra.run.smithi051.stderr:No --period given, using current period=a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9 2020-11-09T18:13:30.455 INFO:teuthology.orchestra.run.smithi051.stdout:[ 2020-11-09T18:13:30.455 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:30.456 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945610.148613_225.1", 2020-11-09T18:13:30.456 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "2020-11-09T18:13:30.148613Z" 2020-11-09T18:13:30.456 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:30.456 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:30.456 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:13:30.456 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "2020-11-09T18:13:02.432939Z" 2020-11-09T18:13:30.456 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:30.457 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:30.457 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:13:30.457 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:13:30.457 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:30.457 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:30.457 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:13:30.458 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:13:30.458 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:30.458 INFO:teuthology.orchestra.run.smithi051.stdout:] 2020-11-09T18:13:30.459 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-09T18:13:30.460 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:30.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:30.460 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:13:30.501 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:13:30.501 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:13:30.742 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:13:30.742 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:13:30.743 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:13:30.743 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:13:30.743 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:13:30.743 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:13:30.743 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:30.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:30.744 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:13:30.744 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:30.744 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:13:30.744 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:30.745 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:30.745 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945610.148613_225.1", 2020-11-09T18:13:30.745 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:30.745 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:13:30.746 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:30.746 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:30.148613Z", 2020-11-09T18:13:30.746 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:30.746 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:30.746 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:30.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:30.747 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:13:30.747 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:30.747 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:30.748 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:13:30.748 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:30.748 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:30.748 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:30.748 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:13:30.749 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:30.749 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:30.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:30.749 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:30.750 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:13:30.750 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:30.750 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:30.750 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:13:30.750 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:30.751 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:30.751 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:30.751 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:13:30.751 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:30.751 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:30.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:30.752 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:30.752 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:13:30.752 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:30.752 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:30.752 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:13:30.752 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:30.753 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:30.753 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:30.753 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:13:30.753 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:30.753 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:30.753 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:30.753 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:13:30.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:30.754 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:13:30.754 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:13:30.754 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:13:30.754 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:30.754 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:13:30.755 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-09T18:13:30.756 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-09T18:13:30.756 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:30.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:30.757 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:13:30.798 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:13:30.799 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:13:31.052 INFO:teuthology.orchestra.run.smithi051.stdout:{ 2020-11-09T18:13:31.053 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_status": { 2020-11-09T18:13:31.053 INFO:teuthology.orchestra.run.smithi051.stdout: "info": { 2020-11-09T18:13:31.053 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "sync", 2020-11-09T18:13:31.053 INFO:teuthology.orchestra.run.smithi051.stdout: "num_shards": 4, 2020-11-09T18:13:31.053 INFO:teuthology.orchestra.run.smithi051.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:13:31.054 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:13:31.054 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:31.054 INFO:teuthology.orchestra.run.smithi051.stdout: "markers": [ 2020-11-09T18:13:31.054 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:31.054 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 0, 2020-11-09T18:13:31.054 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:13:31.054 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:13:31.055 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945610.148613_225.1", 2020-11-09T18:13:31.055 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:13:31.055 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 1, 2020-11-09T18:13:31.055 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:13:31.055 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:30.148613Z", 2020-11-09T18:13:31.055 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:13:31.056 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:31.056 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:31.056 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:31.056 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 1, 2020-11-09T18:13:31.056 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:13:31.056 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:13:31.056 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:13:31.056 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:13:31.057 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:13:31.057 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:13:31.057 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:13:31.057 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:13:31.057 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:31.057 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:31.057 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:31.058 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 2, 2020-11-09T18:13:31.058 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:13:31.058 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:13:31.058 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:13:31.058 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:13:31.058 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:13:31.062 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:13:31.062 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:13:31.062 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:13:31.062 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:31.062 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:31.063 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:31.063 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 3, 2020-11-09T18:13:31.063 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:13:31.063 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:13:31.063 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:13:31.063 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:13:31.064 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:13:31.064 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:13:31.064 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:13:31.064 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:13:31.064 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:31.064 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:31.064 INFO:teuthology.orchestra.run.smithi051.stdout: ] 2020-11-09T18:13:31.065 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:31.065 INFO:teuthology.orchestra.run.smithi051.stdout: "full_sync": { 2020-11-09T18:13:31.065 INFO:teuthology.orchestra.run.smithi051.stdout: "total": 1, 2020-11-09T18:13:31.065 INFO:teuthology.orchestra.run.smithi051.stdout: "complete": 1 2020-11-09T18:13:31.065 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:31.065 INFO:teuthology.orchestra.run.smithi051.stdout:} 2020-11-09T18:13:31.067 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-09T18:13:31.067 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-09T18:13:31.067 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:31.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:31.068 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:13:31.114 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:13:31.115 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:13:31.380 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:13:31.380 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:13:31.380 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:13:31.380 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:13:31.380 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:13:31.381 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:13:31.381 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:31.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:31.381 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:13:31.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:31.381 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:13:31.381 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:31.382 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:31.382 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945610.148613_225.1", 2020-11-09T18:13:31.382 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:31.382 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:13:31.382 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:31.382 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:30.148613Z", 2020-11-09T18:13:31.382 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:31.383 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:31.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:31.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:31.383 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:13:31.383 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:31.383 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:31.383 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:13:31.383 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:31.384 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:31.384 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:31.384 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:13:31.384 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:31.384 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:31.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:31.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:31.385 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:13:31.385 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:31.385 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:31.385 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:13:31.385 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:31.385 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:31.385 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:31.386 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:13:31.386 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:31.386 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:31.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:31.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:31.386 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:13:31.386 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:31.387 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:31.387 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:13:31.387 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:31.387 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:31.387 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:31.387 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:13:31.387 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:31.387 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:31.388 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:31.389 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:13:31.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:31.389 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:13:31.389 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:13:31.389 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:13:31.390 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:31.390 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:13:31.391 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-09T18:13:31.687 INFO:tasks.rgw.c1.client.0.smithi051.stdout:Parse error at line 1: 2020-11-09T18:13:31.687 INFO:tasks.rgw.c1.client.0.smithi051.stdout:syntax error 2020-11-09T18:13:31.731 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_delete_notempty ... ERROR 2020-11-09T18:13:31.732 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ymzeui-39 2020-11-09T18:13:31.776 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ymzeui-40 2020-11-09T18:13:32.566 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ymzeui-41 2020-11-09T18:13:32.675 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:32.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:32.677 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:13:32.713 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:13:32.714 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:13:32.927 INFO:teuthology.orchestra.run.smithi051.stderr:No --period given, using current period=a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9 2020-11-09T18:13:32.960 INFO:teuthology.orchestra.run.smithi051.stdout:[ 2020-11-09T18:13:32.960 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:32.960 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945612.653005_239.1", 2020-11-09T18:13:32.960 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "2020-11-09T18:13:32.653005Z" 2020-11-09T18:13:32.960 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:32.961 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:32.961 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:13:32.961 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "2020-11-09T18:13:02.432939Z" 2020-11-09T18:13:32.961 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:32.961 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:32.961 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:13:32.962 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:13:32.962 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:32.962 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:32.962 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:13:32.962 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:13:32.962 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:32.962 INFO:teuthology.orchestra.run.smithi051.stdout:] 2020-11-09T18:13:32.964 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-09T18:13:32.966 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:32.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:32.968 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:13:33.010 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:13:33.011 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:13:33.258 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:13:33.259 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:13:33.259 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:13:33.259 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:13:33.259 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:13:33.259 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:13:33.260 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:33.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:33.260 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:13:33.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:33.260 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:13:33.260 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:33.260 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:33.261 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945612.653005_239.1", 2020-11-09T18:13:33.261 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:33.261 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:13:33.261 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:33.261 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:32.653005Z", 2020-11-09T18:13:33.261 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:33.261 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:33.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:33.262 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:33.262 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:13:33.262 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:33.262 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:33.262 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:13:33.262 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:33.263 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:33.272 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:33.272 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:13:33.272 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:33.272 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:33.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:33.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:33.273 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:13:33.273 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:33.273 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:33.273 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:13:33.273 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:33.274 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:33.274 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:33.274 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:13:33.274 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:33.274 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:33.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:33.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:33.275 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:13:33.275 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:33.275 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:33.275 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:13:33.275 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:33.275 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:33.276 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:33.276 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:13:33.276 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:33.276 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:33.276 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:33.276 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:13:33.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:33.277 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:13:33.277 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:13:33.277 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:13:33.277 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:33.277 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:13:33.278 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-09T18:13:33.279 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-09T18:13:33.279 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:33.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:33.281 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:13:33.327 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:13:33.327 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:13:33.574 INFO:teuthology.orchestra.run.smithi051.stdout:{ 2020-11-09T18:13:33.575 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_status": { 2020-11-09T18:13:33.575 INFO:teuthology.orchestra.run.smithi051.stdout: "info": { 2020-11-09T18:13:33.575 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "sync", 2020-11-09T18:13:33.575 INFO:teuthology.orchestra.run.smithi051.stdout: "num_shards": 4, 2020-11-09T18:13:33.575 INFO:teuthology.orchestra.run.smithi051.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:13:33.576 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:13:33.576 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:33.576 INFO:teuthology.orchestra.run.smithi051.stdout: "markers": [ 2020-11-09T18:13:33.576 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:33.576 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 0, 2020-11-09T18:13:33.576 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:13:33.577 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:13:33.577 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945612.653005_239.1", 2020-11-09T18:13:33.577 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:13:33.577 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 1, 2020-11-09T18:13:33.579 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:13:33.579 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:32.653005Z", 2020-11-09T18:13:33.580 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:13:33.580 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:33.580 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:33.580 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:33.580 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 1, 2020-11-09T18:13:33.580 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:13:33.581 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:13:33.581 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:13:33.581 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:13:33.581 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:13:33.581 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:13:33.581 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:13:33.581 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:13:33.582 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:33.582 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:33.582 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:33.582 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 2, 2020-11-09T18:13:33.582 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:13:33.582 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:13:33.582 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:13:33.583 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:13:33.583 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:13:33.583 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:13:33.583 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:13:33.583 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:13:33.583 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:33.583 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:33.584 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:13:33.584 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 3, 2020-11-09T18:13:33.584 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:13:33.584 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:13:33.584 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:13:33.584 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:13:33.584 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:13:33.584 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:13:33.585 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:13:33.585 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:13:33.585 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:33.585 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:33.585 INFO:teuthology.orchestra.run.smithi051.stdout: ] 2020-11-09T18:13:33.585 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:13:33.586 INFO:teuthology.orchestra.run.smithi051.stdout: "full_sync": { 2020-11-09T18:13:33.589 INFO:teuthology.orchestra.run.smithi051.stdout: "total": 1, 2020-11-09T18:13:33.589 INFO:teuthology.orchestra.run.smithi051.stdout: "complete": 1 2020-11-09T18:13:33.589 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:13:33.589 INFO:teuthology.orchestra.run.smithi051.stdout:} 2020-11-09T18:13:33.590 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-09T18:13:33.591 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-09T18:13:33.591 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:33.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:13:33.592 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:13:33.628 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:13:33.629 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:13:33.889 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:13:33.890 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:13:33.890 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:13:33.890 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:13:33.890 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:13:33.891 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:13:33.891 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:33.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:33.891 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:13:33.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:33.891 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:13:33.892 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:33.892 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:33.892 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945612.653005_239.1", 2020-11-09T18:13:33.892 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:33.892 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:13:33.893 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:33.893 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:32.653005Z", 2020-11-09T18:13:33.893 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:33.893 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:33.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:33.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:33.894 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:13:33.895 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:33.895 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:33.895 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:13:33.895 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:33.895 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:33.896 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:33.896 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:13:33.896 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:33.896 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:33.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:33.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:33.898 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:13:33.898 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:33.898 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:33.898 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:13:33.899 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:33.899 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:33.899 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:33.899 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:13:33.899 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:33.900 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:33.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:33.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:33.900 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:13:33.900 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:33.901 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:13:33.901 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:13:33.901 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:13:33.901 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:13:33.902 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:13:33.902 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:13:33.902 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:13:33.903 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:33.903 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:33.903 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:13:33.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:33.904 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:13:33.904 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:13:33.904 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:13:33.906 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:33.906 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:13:33.907 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-09T18:13:33.908 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2020-11-09T18:13:33.909 INFO:teuthology.orchestra.run:waiting for 300 2020-11-09T18:13:33.910 INFO:tasks.rgw.c1.client.1.smithi051.stdout:2020-11-09T18:13:33.913+0000 7fbb62ffd700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 19144) UID: 0 2020-11-09T18:13:33.910 INFO:tasks.rgw.c1.client.1.smithi051.stdout:2020-11-09T18:13:33.913+0000 7fbb7f090100 -1 shutting down 2020-11-09T18:13:39.513 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2020-11-09T18:13:40.011 INFO:tasks.rgw.c1.client.1:Stopped 2020-11-09T18:13:40.012 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:13:40.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:13:40.013 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-09T18:13:40.052 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:13:40.053 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:13:40.079 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:13:40.079+0000 7f157ddbf0c0 0 NOTICE: overriding master zone: 11f516a7-3288-49a6-9a89-b45ac1dae91a 2020-11-09T18:13:40.079 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:13:40.079+0000 7f157ddbf0c0 0 failed reading obj info from .rgw.root:zone_info.11f516a7-3288-49a6-9a89-b45ac1dae91a: (2) No such file or directory 2020-11-09T18:13:40.080 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:13:40.079+0000 7f157ddbf0c0 0 WARNING: could not read zone params for zone id=11f516a7-3288-49a6-9a89-b45ac1dae91a name=test-zone1 2020-11-09T18:13:40.083 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:13:40.083+0000 7f157ddbf0c0 0 failed reading obj info from .rgw.root:zone_info.a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6: (2) No such file or directory 2020-11-09T18:13:40.083 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:13:40.083+0000 7f157ddbf0c0 0 WARNING: could not read zone params for zone id=a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6 name=test-zone3 2020-11-09T18:13:40.094 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:13:40.094 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc", 2020-11-09T18:13:40.094 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "test-zone2", 2020-11-09T18:13:40.094 INFO:teuthology.orchestra.run.smithi078.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-11-09T18:13:40.095 INFO:teuthology.orchestra.run.smithi078.stdout: "control_pool": "test-zone2.rgw.control", 2020-11-09T18:13:40.095 INFO:teuthology.orchestra.run.smithi078.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-11-09T18:13:40.095 INFO:teuthology.orchestra.run.smithi078.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-11-09T18:13:40.095 INFO:teuthology.orchestra.run.smithi078.stdout: "log_pool": "test-zone2.rgw.log", 2020-11-09T18:13:40.095 INFO:teuthology.orchestra.run.smithi078.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-11-09T18:13:40.096 INFO:teuthology.orchestra.run.smithi078.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-11-09T18:13:40.096 INFO:teuthology.orchestra.run.smithi078.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-11-09T18:13:40.096 INFO:teuthology.orchestra.run.smithi078.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-11-09T18:13:40.096 INFO:teuthology.orchestra.run.smithi078.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-11-09T18:13:40.096 INFO:teuthology.orchestra.run.smithi078.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-11-09T18:13:40.096 INFO:teuthology.orchestra.run.smithi078.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-11-09T18:13:40.096 INFO:teuthology.orchestra.run.smithi078.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-11-09T18:13:40.097 INFO:teuthology.orchestra.run.smithi078.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-11-09T18:13:40.097 INFO:teuthology.orchestra.run.smithi078.stdout: "system_key": { 2020-11-09T18:13:40.097 INFO:teuthology.orchestra.run.smithi078.stdout: "access_key": "ZAW7MMQ493TT1LMM", 2020-11-09T18:13:40.097 INFO:teuthology.orchestra.run.smithi078.stdout: "secret_key": "HQHAA9hTQSSDkU0yH5qgAQS7JzFsECDH" 2020-11-09T18:13:40.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:40.097 INFO:teuthology.orchestra.run.smithi078.stdout: "placement_pools": [ 2020-11-09T18:13:40.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:40.098 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "default-placement", 2020-11-09T18:13:40.098 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:13:40.098 INFO:teuthology.orchestra.run.smithi078.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-11-09T18:13:40.098 INFO:teuthology.orchestra.run.smithi078.stdout: "storage_classes": { 2020-11-09T18:13:40.098 INFO:teuthology.orchestra.run.smithi078.stdout: "STANDARD": { 2020-11-09T18:13:40.098 INFO:teuthology.orchestra.run.smithi078.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2020-11-09T18:13:40.098 INFO:teuthology.orchestra.run.smithi078.stdout: "compression_type": "random" 2020-11-09T18:13:40.099 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:40.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:40.099 INFO:teuthology.orchestra.run.smithi078.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-11-09T18:13:40.099 INFO:teuthology.orchestra.run.smithi078.stdout: "index_type": 0 2020-11-09T18:13:40.099 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:40.099 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:40.099 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:13:40.100 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_id": "0de9c5fb-e243-4abd-b414-59ffbc2cb575", 2020-11-09T18:13:40.100 INFO:teuthology.orchestra.run.smithi078.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2020-11-09T18:13:40.100 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:13:40.101 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:13:40.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:13:40.102 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-09T18:13:40.183 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:13:40.183 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:13:40.498 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:13:40.498 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "bbf9f070-21b9-4940-95e9-d031ba1c647f", 2020-11-09T18:13:40.499 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2020-11-09T18:13:40.499 INFO:teuthology.orchestra.run.smithi078.stdout: "predecessor_uuid": "a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9", 2020-11-09T18:13:40.499 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": [ 2020-11-09T18:13:40.499 INFO:teuthology.orchestra.run.smithi078.stdout: "1_1604945612.653005_239.1", 2020-11-09T18:13:40.500 INFO:teuthology.orchestra.run.smithi078.stdout: "1_1604945582.432939_95.1", 2020-11-09T18:13:40.500 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2020-11-09T18:13:40.501 INFO:teuthology.orchestra.run.smithi078.stdout: "" 2020-11-09T18:13:40.501 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:13:40.501 INFO:teuthology.orchestra.run.smithi078.stdout: "period_map": { 2020-11-09T18:13:40.501 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "bbf9f070-21b9-4940-95e9-d031ba1c647f", 2020-11-09T18:13:40.502 INFO:teuthology.orchestra.run.smithi078.stdout: "zonegroups": [ 2020-11-09T18:13:40.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:40.502 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "test-zonegroup", 2020-11-09T18:13:40.502 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "test-zonegroup", 2020-11-09T18:13:40.502 INFO:teuthology.orchestra.run.smithi078.stdout: "api_name": "", 2020-11-09T18:13:40.502 INFO:teuthology.orchestra.run.smithi078.stdout: "is_master": "true", 2020-11-09T18:13:40.502 INFO:teuthology.orchestra.run.smithi078.stdout: "endpoints": [ 2020-11-09T18:13:40.503 INFO:teuthology.orchestra.run.smithi078.stdout: "http://smithi051.front.sepia.ceph.com:8000" 2020-11-09T18:13:40.503 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:13:40.503 INFO:teuthology.orchestra.run.smithi078.stdout: "hostnames": [], 2020-11-09T18:13:40.503 INFO:teuthology.orchestra.run.smithi078.stdout: "hostnames_s3website": [], 2020-11-09T18:13:40.503 INFO:teuthology.orchestra.run.smithi078.stdout: "master_zone": "4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc", 2020-11-09T18:13:40.503 INFO:teuthology.orchestra.run.smithi078.stdout: "zones": [ 2020-11-09T18:13:40.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:40.504 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:13:40.504 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "test-zone1", 2020-11-09T18:13:40.504 INFO:teuthology.orchestra.run.smithi078.stdout: "endpoints": [ 2020-11-09T18:13:40.504 INFO:teuthology.orchestra.run.smithi078.stdout: "http://smithi051.front.sepia.ceph.com:8000" 2020-11-09T18:13:40.504 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:13:40.504 INFO:teuthology.orchestra.run.smithi078.stdout: "log_meta": "false", 2020-11-09T18:13:40.504 INFO:teuthology.orchestra.run.smithi078.stdout: "log_data": "true", 2020-11-09T18:13:40.504 INFO:teuthology.orchestra.run.smithi078.stdout: "bucket_index_max_shards": 11, 2020-11-09T18:13:40.505 INFO:teuthology.orchestra.run.smithi078.stdout: "read_only": "false", 2020-11-09T18:13:40.505 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_type": "", 2020-11-09T18:13:40.505 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_from_all": "true", 2020-11-09T18:13:40.505 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_from": [], 2020-11-09T18:13:40.505 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect_zone": "" 2020-11-09T18:13:40.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:40.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:40.506 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "4600c141-233a-40b8-a0db-e3e3a34df6c3", 2020-11-09T18:13:40.506 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "test-zone4", 2020-11-09T18:13:40.506 INFO:teuthology.orchestra.run.smithi078.stdout: "endpoints": [ 2020-11-09T18:13:40.506 INFO:teuthology.orchestra.run.smithi078.stdout: "http://smithi078.front.sepia.ceph.com:8001" 2020-11-09T18:13:40.506 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:13:40.506 INFO:teuthology.orchestra.run.smithi078.stdout: "log_meta": "false", 2020-11-09T18:13:40.506 INFO:teuthology.orchestra.run.smithi078.stdout: "log_data": "true", 2020-11-09T18:13:40.507 INFO:teuthology.orchestra.run.smithi078.stdout: "bucket_index_max_shards": 11, 2020-11-09T18:13:40.507 INFO:teuthology.orchestra.run.smithi078.stdout: "read_only": "false", 2020-11-09T18:13:40.507 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_type": "pubsub", 2020-11-09T18:13:40.507 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_from_all": "false", 2020-11-09T18:13:40.507 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_from": [ 2020-11-09T18:13:40.507 INFO:teuthology.orchestra.run.smithi078.stdout: "test-zone1" 2020-11-09T18:13:40.507 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:13:40.508 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect_zone": "" 2020-11-09T18:13:40.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:40.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:40.508 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc", 2020-11-09T18:13:40.508 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "test-zone2", 2020-11-09T18:13:40.508 INFO:teuthology.orchestra.run.smithi078.stdout: "endpoints": [ 2020-11-09T18:13:40.508 INFO:teuthology.orchestra.run.smithi078.stdout: "http://smithi078.front.sepia.ceph.com:8000" 2020-11-09T18:13:40.508 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:13:40.509 INFO:teuthology.orchestra.run.smithi078.stdout: "log_meta": "false", 2020-11-09T18:13:40.509 INFO:teuthology.orchestra.run.smithi078.stdout: "log_data": "true", 2020-11-09T18:13:40.509 INFO:teuthology.orchestra.run.smithi078.stdout: "bucket_index_max_shards": 11, 2020-11-09T18:13:40.509 INFO:teuthology.orchestra.run.smithi078.stdout: "read_only": "false", 2020-11-09T18:13:40.509 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_type": "", 2020-11-09T18:13:40.509 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_from_all": "true", 2020-11-09T18:13:40.509 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_from": [], 2020-11-09T18:13:40.510 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect_zone": "" 2020-11-09T18:13:40.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:40.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:40.512 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6", 2020-11-09T18:13:40.512 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "test-zone3", 2020-11-09T18:13:40.512 INFO:teuthology.orchestra.run.smithi078.stdout: "endpoints": [ 2020-11-09T18:13:40.512 INFO:teuthology.orchestra.run.smithi078.stdout: "http://smithi051.front.sepia.ceph.com:8001" 2020-11-09T18:13:40.512 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:13:40.512 INFO:teuthology.orchestra.run.smithi078.stdout: "log_meta": "false", 2020-11-09T18:13:40.512 INFO:teuthology.orchestra.run.smithi078.stdout: "log_data": "true", 2020-11-09T18:13:40.513 INFO:teuthology.orchestra.run.smithi078.stdout: "bucket_index_max_shards": 11, 2020-11-09T18:13:40.513 INFO:teuthology.orchestra.run.smithi078.stdout: "read_only": "false", 2020-11-09T18:13:40.513 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_type": "", 2020-11-09T18:13:40.513 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_from_all": "true", 2020-11-09T18:13:40.513 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_from": [], 2020-11-09T18:13:40.513 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect_zone": "" 2020-11-09T18:13:40.513 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:40.513 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:13:40.514 INFO:teuthology.orchestra.run.smithi078.stdout: "placement_targets": [ 2020-11-09T18:13:40.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:40.514 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "default-placement", 2020-11-09T18:13:40.514 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2020-11-09T18:13:40.514 INFO:teuthology.orchestra.run.smithi078.stdout: "storage_classes": [ 2020-11-09T18:13:40.514 INFO:teuthology.orchestra.run.smithi078.stdout: "STANDARD" 2020-11-09T18:13:40.515 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:13:40.515 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:40.515 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:13:40.515 INFO:teuthology.orchestra.run.smithi078.stdout: "default_placement": "default-placement", 2020-11-09T18:13:40.515 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_id": "0de9c5fb-e243-4abd-b414-59ffbc2cb575", 2020-11-09T18:13:40.515 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_policy": { 2020-11-09T18:13:40.515 INFO:teuthology.orchestra.run.smithi078.stdout: "groups": [] 2020-11-09T18:13:40.515 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:40.516 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:40.516 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:13:40.518 INFO:teuthology.orchestra.run.smithi078.stdout: "short_zone_ids": [ 2020-11-09T18:13:40.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:40.519 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:13:40.519 INFO:teuthology.orchestra.run.smithi078.stdout: "val": 2019612975 2020-11-09T18:13:40.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:40.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:40.519 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "4600c141-233a-40b8-a0db-e3e3a34df6c3", 2020-11-09T18:13:40.519 INFO:teuthology.orchestra.run.smithi078.stdout: "val": 1720005997 2020-11-09T18:13:40.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:40.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:40.520 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc", 2020-11-09T18:13:40.520 INFO:teuthology.orchestra.run.smithi078.stdout: "val": 2281443694 2020-11-09T18:13:40.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:40.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:13:40.523 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6", 2020-11-09T18:13:40.523 INFO:teuthology.orchestra.run.smithi078.stdout: "val": 648291153 2020-11-09T18:13:40.523 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:40.523 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:13:40.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:40.524 INFO:teuthology.orchestra.run.smithi078.stdout: "master_zonegroup": "test-zonegroup", 2020-11-09T18:13:40.524 INFO:teuthology.orchestra.run.smithi078.stdout: "master_zone": "4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc", 2020-11-09T18:13:40.524 INFO:teuthology.orchestra.run.smithi078.stdout: "period_config": { 2020-11-09T18:13:40.524 INFO:teuthology.orchestra.run.smithi078.stdout: "bucket_quota": { 2020-11-09T18:13:40.524 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": false, 2020-11-09T18:13:40.524 INFO:teuthology.orchestra.run.smithi078.stdout: "check_on_raw": false, 2020-11-09T18:13:40.525 INFO:teuthology.orchestra.run.smithi078.stdout: "max_size": -1, 2020-11-09T18:13:40.525 INFO:teuthology.orchestra.run.smithi078.stdout: "max_size_kb": 0, 2020-11-09T18:13:40.525 INFO:teuthology.orchestra.run.smithi078.stdout: "max_objects": -1 2020-11-09T18:13:40.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:40.525 INFO:teuthology.orchestra.run.smithi078.stdout: "user_quota": { 2020-11-09T18:13:40.525 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": false, 2020-11-09T18:13:40.525 INFO:teuthology.orchestra.run.smithi078.stdout: "check_on_raw": false, 2020-11-09T18:13:40.526 INFO:teuthology.orchestra.run.smithi078.stdout: "max_size": -1, 2020-11-09T18:13:40.526 INFO:teuthology.orchestra.run.smithi078.stdout: "max_size_kb": 0, 2020-11-09T18:13:40.526 INFO:teuthology.orchestra.run.smithi078.stdout: "max_objects": -1 2020-11-09T18:13:40.526 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:13:40.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:13:40.526 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_id": "0de9c5fb-e243-4abd-b414-59ffbc2cb575", 2020-11-09T18:13:40.526 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_name": "test-realm", 2020-11-09T18:13:40.527 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 3 2020-11-09T18:13:40.527 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:13:40.528 INFO:rgw_multi.tests:Set master zone=test-zone2, waiting 60s for reconfiguration.. 2020-11-09T18:13:45.129 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2020-11-09T18:14:40.529 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ymzeui-42 2020-11-09T18:14:40.600 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ymzeui-43 2020-11-09T18:14:40.637 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:14:40.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:14:40.639 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:14:40.673 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:14:40.674 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:14:40.884 INFO:teuthology.orchestra.run.smithi078.stderr:No --period given, using current period=bbf9f070-21b9-4940-95e9-d031ba1c647f 2020-11-09T18:14:40.916 INFO:teuthology.orchestra.run.smithi078.stdout:[ 2020-11-09T18:14:40.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:14:40.917 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945680.637050_159.1", 2020-11-09T18:14:40.917 INFO:teuthology.orchestra.run.smithi078.stdout: "last_update": "2020-11-09T18:14:40.637050Z" 2020-11-09T18:14:40.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:14:40.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:14:40.918 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:14:40.918 INFO:teuthology.orchestra.run.smithi078.stdout: "last_update": "0.000000" 2020-11-09T18:14:40.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:14:40.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:14:40.919 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:14:40.919 INFO:teuthology.orchestra.run.smithi078.stdout: "last_update": "0.000000" 2020-11-09T18:14:40.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:14:40.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:14:40.919 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:14:40.919 INFO:teuthology.orchestra.run.smithi078.stdout: "last_update": "0.000000" 2020-11-09T18:14:40.920 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:14:40.920 INFO:teuthology.orchestra.run.smithi078.stdout:] 2020-11-09T18:14:40.921 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone1 2020-11-09T18:14:40.921 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:14:40.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:14:40.922 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:14:40.963 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:14:40.963 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:14:41.213 INFO:teuthology.orchestra.run.smithi051.stdout:{ 2020-11-09T18:14:41.214 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_status": { 2020-11-09T18:14:41.214 INFO:teuthology.orchestra.run.smithi051.stdout: "info": { 2020-11-09T18:14:41.214 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "sync", 2020-11-09T18:14:41.214 INFO:teuthology.orchestra.run.smithi051.stdout: "num_shards": 4, 2020-11-09T18:14:41.214 INFO:teuthology.orchestra.run.smithi051.stdout: "period": "bbf9f070-21b9-4940-95e9-d031ba1c647f", 2020-11-09T18:14:41.215 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 3 2020-11-09T18:14:41.215 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:14:41.215 INFO:teuthology.orchestra.run.smithi051.stdout: "markers": [ 2020-11-09T18:14:41.215 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:14:41.215 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 0, 2020-11-09T18:14:41.215 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:14:41.215 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:14:41.216 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945680.637050_159.1", 2020-11-09T18:14:41.216 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:14:41.216 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 80, 2020-11-09T18:14:41.216 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:14:41.216 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:14:40.637050Z", 2020-11-09T18:14:41.216 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 3 2020-11-09T18:14:41.216 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:14:41.217 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:14:41.217 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:14:41.217 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 1, 2020-11-09T18:14:41.217 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:14:41.217 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:14:41.217 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:14:41.217 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:14:41.218 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:14:41.218 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:14:41.218 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:14:41.218 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 3 2020-11-09T18:14:41.218 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:14:41.218 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:14:41.218 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:14:41.219 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 2, 2020-11-09T18:14:41.219 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:14:41.219 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:14:41.219 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:14:41.219 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:14:41.219 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:14:41.219 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:14:41.220 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:14:41.220 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 3 2020-11-09T18:14:41.220 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:14:41.220 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:14:41.220 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:14:41.220 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 3, 2020-11-09T18:14:41.220 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:14:41.220 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:14:41.221 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:14:41.221 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:14:41.221 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:14:41.221 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:14:41.221 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:14:41.221 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 3 2020-11-09T18:14:41.221 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:14:41.222 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:14:41.222 INFO:teuthology.orchestra.run.smithi051.stdout: ] 2020-11-09T18:14:41.222 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:14:41.222 INFO:teuthology.orchestra.run.smithi051.stdout: "full_sync": { 2020-11-09T18:14:41.222 INFO:teuthology.orchestra.run.smithi051.stdout: "total": 80, 2020-11-09T18:14:41.222 INFO:teuthology.orchestra.run.smithi051.stdout: "complete": 80 2020-11-09T18:14:41.222 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:14:41.223 INFO:teuthology.orchestra.run.smithi051.stdout:} 2020-11-09T18:14:41.224 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone1 2020-11-09T18:14:41.224 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:14:41.225 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:14:41.225 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-09T18:14:41.308 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:14:41.309 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:14:41.333 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:14:41.334+0000 7facf4ab90c0 0 NOTICE: overriding master zone: 4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc 2020-11-09T18:14:41.336 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:14:41.338+0000 7facf4ab90c0 0 failed reading obj info from .rgw.root:zone_info.4600c141-233a-40b8-a0db-e3e3a34df6c3: (2) No such file or directory 2020-11-09T18:14:41.336 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:14:41.338+0000 7facf4ab90c0 0 WARNING: could not read zone params for zone id=4600c141-233a-40b8-a0db-e3e3a34df6c3 name=test-zone4 2020-11-09T18:14:41.336 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:14:41.338+0000 7facf4ab90c0 0 failed reading obj info from .rgw.root:zone_info.4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc: (2) No such file or directory 2020-11-09T18:14:41.336 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:14:41.338+0000 7facf4ab90c0 0 WARNING: could not read zone params for zone id=4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc name=test-zone2 2020-11-09T18:14:41.350 INFO:teuthology.orchestra.run.smithi051.stdout:{ 2020-11-09T18:14:41.350 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:14:41.350 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "test-zone1", 2020-11-09T18:14:41.350 INFO:teuthology.orchestra.run.smithi051.stdout: "domain_root": "test-zone1.rgw.meta:root", 2020-11-09T18:14:41.351 INFO:teuthology.orchestra.run.smithi051.stdout: "control_pool": "test-zone1.rgw.control", 2020-11-09T18:14:41.351 INFO:teuthology.orchestra.run.smithi051.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2020-11-09T18:14:41.351 INFO:teuthology.orchestra.run.smithi051.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2020-11-09T18:14:41.351 INFO:teuthology.orchestra.run.smithi051.stdout: "log_pool": "test-zone1.rgw.log", 2020-11-09T18:14:41.351 INFO:teuthology.orchestra.run.smithi051.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2020-11-09T18:14:41.351 INFO:teuthology.orchestra.run.smithi051.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2020-11-09T18:14:41.351 INFO:teuthology.orchestra.run.smithi051.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2020-11-09T18:14:41.352 INFO:teuthology.orchestra.run.smithi051.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2020-11-09T18:14:41.352 INFO:teuthology.orchestra.run.smithi051.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2020-11-09T18:14:41.352 INFO:teuthology.orchestra.run.smithi051.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2020-11-09T18:14:41.352 INFO:teuthology.orchestra.run.smithi051.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2020-11-09T18:14:41.352 INFO:teuthology.orchestra.run.smithi051.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2020-11-09T18:14:41.352 INFO:teuthology.orchestra.run.smithi051.stdout: "otp_pool": "test-zone1.rgw.otp", 2020-11-09T18:14:41.352 INFO:teuthology.orchestra.run.smithi051.stdout: "system_key": { 2020-11-09T18:14:41.353 INFO:teuthology.orchestra.run.smithi051.stdout: "access_key": "ZAW7MMQ493TT1LMM", 2020-11-09T18:14:41.353 INFO:teuthology.orchestra.run.smithi051.stdout: "secret_key": "HQHAA9hTQSSDkU0yH5qgAQS7JzFsECDH" 2020-11-09T18:14:41.353 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:14:41.353 INFO:teuthology.orchestra.run.smithi051.stdout: "placement_pools": [ 2020-11-09T18:14:41.353 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:14:41.353 INFO:teuthology.orchestra.run.smithi051.stdout: "key": "default-placement", 2020-11-09T18:14:41.353 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:14:41.354 INFO:teuthology.orchestra.run.smithi051.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2020-11-09T18:14:41.354 INFO:teuthology.orchestra.run.smithi051.stdout: "storage_classes": { 2020-11-09T18:14:41.354 INFO:teuthology.orchestra.run.smithi051.stdout: "STANDARD": { 2020-11-09T18:14:41.354 INFO:teuthology.orchestra.run.smithi051.stdout: "data_pool": "test-zone1.rgw.buckets.data", 2020-11-09T18:14:41.354 INFO:teuthology.orchestra.run.smithi051.stdout: "compression_type": "random" 2020-11-09T18:14:41.354 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:14:41.354 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:14:41.355 INFO:teuthology.orchestra.run.smithi051.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2020-11-09T18:14:41.355 INFO:teuthology.orchestra.run.smithi051.stdout: "index_type": 0 2020-11-09T18:14:41.355 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:14:41.355 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:14:41.355 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:14:41.355 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_id": "0de9c5fb-e243-4abd-b414-59ffbc2cb575", 2020-11-09T18:14:41.355 INFO:teuthology.orchestra.run.smithi051.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2020-11-09T18:14:41.356 INFO:teuthology.orchestra.run.smithi051.stdout:} 2020-11-09T18:14:41.356 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:14:41.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:14:41.357 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-09T18:14:41.438 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:14:41.439 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:14:41.750 INFO:teuthology.orchestra.run.smithi051.stdout:{ 2020-11-09T18:14:41.750 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:14:41.750 INFO:teuthology.orchestra.run.smithi051.stdout: "epoch": 1, 2020-11-09T18:14:41.750 INFO:teuthology.orchestra.run.smithi051.stdout: "predecessor_uuid": "bbf9f070-21b9-4940-95e9-d031ba1c647f", 2020-11-09T18:14:41.751 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_status": [ 2020-11-09T18:14:41.751 INFO:teuthology.orchestra.run.smithi051.stdout: "1_1604945680.637050_159.1", 2020-11-09T18:14:41.751 INFO:teuthology.orchestra.run.smithi051.stdout: "", 2020-11-09T18:14:41.751 INFO:teuthology.orchestra.run.smithi051.stdout: "", 2020-11-09T18:14:41.751 INFO:teuthology.orchestra.run.smithi051.stdout: "" 2020-11-09T18:14:41.751 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:14:41.751 INFO:teuthology.orchestra.run.smithi051.stdout: "period_map": { 2020-11-09T18:14:41.752 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:14:41.752 INFO:teuthology.orchestra.run.smithi051.stdout: "zonegroups": [ 2020-11-09T18:14:41.752 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:14:41.752 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "test-zonegroup", 2020-11-09T18:14:41.752 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "test-zonegroup", 2020-11-09T18:14:41.752 INFO:teuthology.orchestra.run.smithi051.stdout: "api_name": "", 2020-11-09T18:14:41.752 INFO:teuthology.orchestra.run.smithi051.stdout: "is_master": "true", 2020-11-09T18:14:41.753 INFO:teuthology.orchestra.run.smithi051.stdout: "endpoints": [ 2020-11-09T18:14:41.753 INFO:teuthology.orchestra.run.smithi051.stdout: "http://smithi051.front.sepia.ceph.com:8000" 2020-11-09T18:14:41.753 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:14:41.753 INFO:teuthology.orchestra.run.smithi051.stdout: "hostnames": [], 2020-11-09T18:14:41.753 INFO:teuthology.orchestra.run.smithi051.stdout: "hostnames_s3website": [], 2020-11-09T18:14:41.753 INFO:teuthology.orchestra.run.smithi051.stdout: "master_zone": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:14:41.753 INFO:teuthology.orchestra.run.smithi051.stdout: "zones": [ 2020-11-09T18:14:41.754 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:14:41.754 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:14:41.754 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "test-zone1", 2020-11-09T18:14:41.754 INFO:teuthology.orchestra.run.smithi051.stdout: "endpoints": [ 2020-11-09T18:14:41.754 INFO:teuthology.orchestra.run.smithi051.stdout: "http://smithi051.front.sepia.ceph.com:8000" 2020-11-09T18:14:41.754 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:14:41.755 INFO:teuthology.orchestra.run.smithi051.stdout: "log_meta": "false", 2020-11-09T18:14:41.755 INFO:teuthology.orchestra.run.smithi051.stdout: "log_data": "true", 2020-11-09T18:14:41.755 INFO:teuthology.orchestra.run.smithi051.stdout: "bucket_index_max_shards": 11, 2020-11-09T18:14:41.755 INFO:teuthology.orchestra.run.smithi051.stdout: "read_only": "false", 2020-11-09T18:14:41.755 INFO:teuthology.orchestra.run.smithi051.stdout: "tier_type": "", 2020-11-09T18:14:41.755 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from_all": "true", 2020-11-09T18:14:41.755 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from": [], 2020-11-09T18:14:41.756 INFO:teuthology.orchestra.run.smithi051.stdout: "redirect_zone": "" 2020-11-09T18:14:41.756 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:14:41.756 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:14:41.756 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "4600c141-233a-40b8-a0db-e3e3a34df6c3", 2020-11-09T18:14:41.756 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "test-zone4", 2020-11-09T18:14:41.757 INFO:teuthology.orchestra.run.smithi051.stdout: "endpoints": [ 2020-11-09T18:14:41.757 INFO:teuthology.orchestra.run.smithi051.stdout: "http://smithi078.front.sepia.ceph.com:8001" 2020-11-09T18:14:41.757 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:14:41.757 INFO:teuthology.orchestra.run.smithi051.stdout: "log_meta": "false", 2020-11-09T18:14:41.758 INFO:teuthology.orchestra.run.smithi051.stdout: "log_data": "true", 2020-11-09T18:14:41.758 INFO:teuthology.orchestra.run.smithi051.stdout: "bucket_index_max_shards": 11, 2020-11-09T18:14:41.758 INFO:teuthology.orchestra.run.smithi051.stdout: "read_only": "false", 2020-11-09T18:14:41.758 INFO:teuthology.orchestra.run.smithi051.stdout: "tier_type": "pubsub", 2020-11-09T18:14:41.758 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from_all": "false", 2020-11-09T18:14:41.758 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from": [ 2020-11-09T18:14:41.759 INFO:teuthology.orchestra.run.smithi051.stdout: "test-zone1" 2020-11-09T18:14:41.759 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:14:41.759 INFO:teuthology.orchestra.run.smithi051.stdout: "redirect_zone": "" 2020-11-09T18:14:41.759 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:14:41.759 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:14:41.760 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc", 2020-11-09T18:14:41.760 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "test-zone2", 2020-11-09T18:14:41.760 INFO:teuthology.orchestra.run.smithi051.stdout: "endpoints": [ 2020-11-09T18:14:41.760 INFO:teuthology.orchestra.run.smithi051.stdout: "http://smithi078.front.sepia.ceph.com:8000" 2020-11-09T18:14:41.760 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:14:41.760 INFO:teuthology.orchestra.run.smithi051.stdout: "log_meta": "false", 2020-11-09T18:14:41.760 INFO:teuthology.orchestra.run.smithi051.stdout: "log_data": "true", 2020-11-09T18:14:41.761 INFO:teuthology.orchestra.run.smithi051.stdout: "bucket_index_max_shards": 11, 2020-11-09T18:14:41.761 INFO:teuthology.orchestra.run.smithi051.stdout: "read_only": "false", 2020-11-09T18:14:41.761 INFO:teuthology.orchestra.run.smithi051.stdout: "tier_type": "", 2020-11-09T18:14:41.761 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from_all": "true", 2020-11-09T18:14:41.761 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from": [], 2020-11-09T18:14:41.761 INFO:teuthology.orchestra.run.smithi051.stdout: "redirect_zone": "" 2020-11-09T18:14:41.761 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:14:41.762 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:14:41.762 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6", 2020-11-09T18:14:41.762 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "test-zone3", 2020-11-09T18:14:41.762 INFO:teuthology.orchestra.run.smithi051.stdout: "endpoints": [ 2020-11-09T18:14:41.762 INFO:teuthology.orchestra.run.smithi051.stdout: "http://smithi051.front.sepia.ceph.com:8001" 2020-11-09T18:14:41.762 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:14:41.762 INFO:teuthology.orchestra.run.smithi051.stdout: "log_meta": "false", 2020-11-09T18:14:41.763 INFO:teuthology.orchestra.run.smithi051.stdout: "log_data": "true", 2020-11-09T18:14:41.763 INFO:teuthology.orchestra.run.smithi051.stdout: "bucket_index_max_shards": 11, 2020-11-09T18:14:41.763 INFO:teuthology.orchestra.run.smithi051.stdout: "read_only": "false", 2020-11-09T18:14:41.763 INFO:teuthology.orchestra.run.smithi051.stdout: "tier_type": "", 2020-11-09T18:14:41.763 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from_all": "true", 2020-11-09T18:14:41.763 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from": [], 2020-11-09T18:14:41.764 INFO:teuthology.orchestra.run.smithi051.stdout: "redirect_zone": "" 2020-11-09T18:14:41.764 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:14:41.764 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:14:41.764 INFO:teuthology.orchestra.run.smithi051.stdout: "placement_targets": [ 2020-11-09T18:14:41.764 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:14:41.764 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "default-placement", 2020-11-09T18:14:41.764 INFO:teuthology.orchestra.run.smithi051.stdout: "tags": [], 2020-11-09T18:14:41.765 INFO:teuthology.orchestra.run.smithi051.stdout: "storage_classes": [ 2020-11-09T18:14:41.765 INFO:teuthology.orchestra.run.smithi051.stdout: "STANDARD" 2020-11-09T18:14:41.766 INFO:teuthology.orchestra.run.smithi051.stdout: ] 2020-11-09T18:14:41.766 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:14:41.766 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:14:41.766 INFO:teuthology.orchestra.run.smithi051.stdout: "default_placement": "default-placement", 2020-11-09T18:14:41.766 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_id": "0de9c5fb-e243-4abd-b414-59ffbc2cb575", 2020-11-09T18:14:41.766 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_policy": { 2020-11-09T18:14:41.766 INFO:teuthology.orchestra.run.smithi051.stdout: "groups": [] 2020-11-09T18:14:41.767 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:14:41.767 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:14:41.767 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:14:41.777 INFO:teuthology.orchestra.run.smithi051.stdout: "short_zone_ids": [ 2020-11-09T18:14:41.777 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:14:41.777 INFO:teuthology.orchestra.run.smithi051.stdout: "key": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:14:41.778 INFO:teuthology.orchestra.run.smithi051.stdout: "val": 2019612975 2020-11-09T18:14:41.778 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:14:41.779 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:14:41.779 INFO:teuthology.orchestra.run.smithi051.stdout: "key": "4600c141-233a-40b8-a0db-e3e3a34df6c3", 2020-11-09T18:14:41.779 INFO:teuthology.orchestra.run.smithi051.stdout: "val": 1720005997 2020-11-09T18:14:41.779 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:14:41.780 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:14:41.780 INFO:teuthology.orchestra.run.smithi051.stdout: "key": "4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc", 2020-11-09T18:14:41.780 INFO:teuthology.orchestra.run.smithi051.stdout: "val": 2281443694 2020-11-09T18:14:41.781 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:14:41.781 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:14:41.782 INFO:teuthology.orchestra.run.smithi051.stdout: "key": "a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6", 2020-11-09T18:14:41.782 INFO:teuthology.orchestra.run.smithi051.stdout: "val": 648291153 2020-11-09T18:14:41.783 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:14:41.783 INFO:teuthology.orchestra.run.smithi051.stdout: ] 2020-11-09T18:14:41.783 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:14:41.783 INFO:teuthology.orchestra.run.smithi051.stdout: "master_zonegroup": "test-zonegroup", 2020-11-09T18:14:41.784 INFO:teuthology.orchestra.run.smithi051.stdout: "master_zone": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:14:41.784 INFO:teuthology.orchestra.run.smithi051.stdout: "period_config": { 2020-11-09T18:14:41.784 INFO:teuthology.orchestra.run.smithi051.stdout: "bucket_quota": { 2020-11-09T18:14:41.785 INFO:teuthology.orchestra.run.smithi051.stdout: "enabled": false, 2020-11-09T18:14:41.785 INFO:teuthology.orchestra.run.smithi051.stdout: "check_on_raw": false, 2020-11-09T18:14:41.785 INFO:teuthology.orchestra.run.smithi051.stdout: "max_size": -1, 2020-11-09T18:14:41.785 INFO:teuthology.orchestra.run.smithi051.stdout: "max_size_kb": 0, 2020-11-09T18:14:41.785 INFO:teuthology.orchestra.run.smithi051.stdout: "max_objects": -1 2020-11-09T18:14:41.786 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:14:41.786 INFO:teuthology.orchestra.run.smithi051.stdout: "user_quota": { 2020-11-09T18:14:41.786 INFO:teuthology.orchestra.run.smithi051.stdout: "enabled": false, 2020-11-09T18:14:41.786 INFO:teuthology.orchestra.run.smithi051.stdout: "check_on_raw": false, 2020-11-09T18:14:41.786 INFO:teuthology.orchestra.run.smithi051.stdout: "max_size": -1, 2020-11-09T18:14:41.786 INFO:teuthology.orchestra.run.smithi051.stdout: "max_size_kb": 0, 2020-11-09T18:14:41.787 INFO:teuthology.orchestra.run.smithi051.stdout: "max_objects": -1 2020-11-09T18:14:41.787 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:14:41.787 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:14:41.787 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_id": "0de9c5fb-e243-4abd-b414-59ffbc2cb575", 2020-11-09T18:14:41.787 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_name": "test-realm", 2020-11-09T18:14:41.787 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 4 2020-11-09T18:14:41.787 INFO:teuthology.orchestra.run.smithi051.stdout:} 2020-11-09T18:14:41.788 INFO:rgw_multi.tests:Set master zone=test-zone1, waiting 60s for reconfiguration.. 2020-11-09T18:15:41.790 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ymzeui-44 2020-11-09T18:15:41.842 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ymzeui-45 2020-11-09T18:15:41.931 INFO:tasks.rgw.c1.client.1:Restarting daemon 2020-11-09T18:15:41.932 INFO:teuthology.orchestra.run.smithi051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2020-11-09T18:15:41.934 INFO:tasks.rgw.c1.client.1:Started 2020-11-09T18:15:41.935 INFO:teuthology.orchestra.run.smithi051:> curl http://smithi051.front.sepia.ceph.com:8001 2020-11-09T18:15:41.953 INFO:teuthology.orchestra.run.smithi051.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-09T18:15:41.954 INFO:teuthology.orchestra.run.smithi051.stderr: Dload Upload Total Spent Left Speed 2020-11-09T18:15:41.956 INFO:teuthology.orchestra.run.smithi051.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi051.front.sepia.ceph.com port 8001: Connection refused 2020-11-09T18:15:41.956 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-11-09T18:15:42.957 INFO:teuthology.orchestra.run.smithi051:> curl http://smithi051.front.sepia.ceph.com:8001 2020-11-09T18:15:42.972 INFO:teuthology.orchestra.run.smithi051.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-09T18:15:42.972 INFO:teuthology.orchestra.run.smithi051.stderr: Dload Upload Total Spent Left Speed 2020-11-09T18:15:42.978 INFO:teuthology.orchestra.run.smithi051.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 35666 0 --:--:-- --:--:-- --:--:-- 35666 2020-11-09T18:15:42.979 INFO:teuthology.orchestra.run.smithi051.stdout:anonymous 2020-11-09T18:15:42.980 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:42.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:42.981 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:15:43.066 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:15:43.066 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:15:43.309 INFO:teuthology.orchestra.run.smithi051.stderr:No --period given, using current period=f6258aba-300a-4d5d-ad93-a7c198a0e0a1 2020-11-09T18:15:43.339 INFO:teuthology.orchestra.run.smithi051.stdout:[ 2020-11-09T18:15:43.340 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:43.340 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945741.900663_174.1", 2020-11-09T18:15:43.340 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "2020-11-09T18:15:41.900663Z" 2020-11-09T18:15:43.340 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:43.340 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:43.340 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:15:43.341 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:15:43.341 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:43.341 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:43.341 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:15:43.341 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:15:43.341 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:43.341 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:43.342 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:15:43.342 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:15:43.342 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:43.342 INFO:teuthology.orchestra.run.smithi051.stdout:] 2020-11-09T18:15:43.343 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-09T18:15:43.344 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:43.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:43.345 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:15:43.379 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:15:43.379 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:15:43.604 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:15:43.605 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:15:43.605 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:15:43.605 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:15:43.606 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:15:43.606 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:15:43.606 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:15:43.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:43.606 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:15:43.607 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:43.607 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:15:43.607 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:15:43.607 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:15:43.607 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945741.900663_174.1", 2020-11-09T18:15:43.607 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:15:43.607 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:15:43.608 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:15:43.608 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:15:41.900663Z", 2020-11-09T18:15:43.608 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:15:43.608 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:43.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:43.608 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:43.609 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:15:43.609 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:15:43.609 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:15:43.609 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:15:43.609 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:15:43.610 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:15:43.610 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:15:43.610 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:15:43.610 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:15:43.610 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:43.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:43.611 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:43.611 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:15:43.611 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:15:43.612 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:15:43.612 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:15:43.612 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:15:43.612 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:15:43.612 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:15:43.612 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:15:43.612 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:15:43.613 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:43.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:43.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:43.613 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:15:43.613 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:15:43.613 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:15:43.614 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:15:43.614 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:15:43.614 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:15:43.614 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:15:43.614 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:15:43.615 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:15:43.615 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:43.615 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:43.615 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:15:43.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:43.615 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:15:43.616 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:15:43.616 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:15:43.616 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:43.616 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:15:43.618 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-09T18:15:43.619 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-09T18:15:43.619 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:43.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:43.621 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:15:43.661 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:15:43.662 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:15:43.911 INFO:teuthology.orchestra.run.smithi051.stdout:{ 2020-11-09T18:15:43.912 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_status": { 2020-11-09T18:15:43.912 INFO:teuthology.orchestra.run.smithi051.stdout: "info": { 2020-11-09T18:15:43.912 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "sync", 2020-11-09T18:15:43.912 INFO:teuthology.orchestra.run.smithi051.stdout: "num_shards": 4, 2020-11-09T18:15:43.912 INFO:teuthology.orchestra.run.smithi051.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:15:43.912 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 4 2020-11-09T18:15:43.912 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:43.913 INFO:teuthology.orchestra.run.smithi051.stdout: "markers": [ 2020-11-09T18:15:43.913 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:43.913 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 0, 2020-11-09T18:15:43.913 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:15:43.913 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:15:43.913 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945741.900663_174.1", 2020-11-09T18:15:43.914 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:15:43.914 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 1, 2020-11-09T18:15:43.914 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:15:43.914 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:15:41.900663Z", 2020-11-09T18:15:43.914 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 4 2020-11-09T18:15:43.914 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:43.914 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:43.915 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:43.915 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 1, 2020-11-09T18:15:43.915 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:15:43.915 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:15:43.915 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:15:43.915 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:15:43.915 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:15:43.915 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:15:43.916 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:15:43.916 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:15:43.916 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:43.916 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:43.916 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:43.916 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 2, 2020-11-09T18:15:43.916 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:15:43.917 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:15:43.917 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:15:43.917 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:15:43.917 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:15:43.917 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:15:43.917 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:15:43.917 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:15:43.918 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:43.918 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:43.918 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:43.918 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 3, 2020-11-09T18:15:43.918 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:15:43.918 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:15:43.918 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:15:43.918 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:15:43.919 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:15:43.919 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:15:43.919 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:15:43.919 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:15:43.919 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:43.919 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:43.919 INFO:teuthology.orchestra.run.smithi051.stdout: ] 2020-11-09T18:15:43.920 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:43.920 INFO:teuthology.orchestra.run.smithi051.stdout: "full_sync": { 2020-11-09T18:15:43.920 INFO:teuthology.orchestra.run.smithi051.stdout: "total": 1, 2020-11-09T18:15:43.920 INFO:teuthology.orchestra.run.smithi051.stdout: "complete": 1 2020-11-09T18:15:43.920 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:43.920 INFO:teuthology.orchestra.run.smithi051.stdout:} 2020-11-09T18:15:43.921 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-09T18:15:43.922 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-09T18:15:43.922 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:43.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:43.923 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:15:43.963 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:15:43.963 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:15:44.229 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:15:44.229 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:15:44.229 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:15:44.230 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:15:44.230 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:15:44.230 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:15:44.230 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:15:44.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:44.230 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:15:44.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:44.231 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:15:44.231 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:15:44.231 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:15:44.231 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945741.900663_174.1", 2020-11-09T18:15:44.231 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:15:44.231 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:15:44.232 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:15:44.232 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:15:41.900663Z", 2020-11-09T18:15:44.232 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:15:44.232 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:44.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:44.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:44.232 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:15:44.233 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:15:44.233 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:15:44.233 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:15:44.233 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:15:44.233 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:15:44.234 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:15:44.234 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:15:44.234 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:15:44.235 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:44.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:44.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:44.235 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:15:44.235 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:15:44.236 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:15:44.236 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:15:44.236 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:15:44.236 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:15:44.236 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:15:44.236 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:15:44.236 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:15:44.236 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:44.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:44.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:44.237 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:15:44.237 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:15:44.237 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:15:44.237 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:15:44.237 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:15:44.238 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:15:44.238 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:15:44.238 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:15:44.238 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:15:44.238 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:44.239 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:44.239 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:15:44.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:44.239 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:15:44.239 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:15:44.240 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:15:44.240 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:44.240 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:15:44.241 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-09T18:15:44.241 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ymzeui-39 zones={test-zone2, test-zone1} 2020-11-09T18:15:44.891 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T18:15:44.901 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T18:15:45.010 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ymzeui-39 zones={test-zone2, test-zone1} 2020-11-09T18:15:45.011 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ymzeui-39 zones={test-zone3, test-zone1} 2020-11-09T18:15:45.033 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T18:15:45.086 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T18:15:45.153 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ymzeui-39 zones={test-zone3, test-zone1} 2020-11-09T18:15:45.154 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ymzeui-39 zones={test-zone1, test-zone2} 2020-11-09T18:15:45.165 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T18:15:45.214 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T18:15:45.277 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ymzeui-39 zones={test-zone1, test-zone2} 2020-11-09T18:15:45.277 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ymzeui-39 zones={test-zone3, test-zone2} 2020-11-09T18:15:45.287 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T18:15:45.334 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T18:15:45.398 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ymzeui-39 zones={test-zone3, test-zone2} 2020-11-09T18:15:45.399 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ymzeui-39 zones={test-zone1, test-zone3} 2020-11-09T18:15:45.413 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T18:15:45.458 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T18:15:45.522 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ymzeui-39 zones={test-zone1, test-zone3} 2020-11-09T18:15:45.523 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ymzeui-39 zones={test-zone2, test-zone3} 2020-11-09T18:15:45.534 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T18:15:45.582 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T18:15:45.645 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ymzeui-39 zones={test-zone2, test-zone3} 2020-11-09T18:15:45.646 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ymzeui-40 zones={test-zone2, test-zone1} 2020-11-09T18:15:45.661 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T18:15:45.710 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T18:15:45.778 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ymzeui-40 zones={test-zone2, test-zone1} 2020-11-09T18:15:45.779 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ymzeui-40 zones={test-zone3, test-zone1} 2020-11-09T18:15:45.792 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T18:15:45.842 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T18:15:45.905 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ymzeui-40 zones={test-zone3, test-zone1} 2020-11-09T18:15:45.907 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ymzeui-40 zones={test-zone1, test-zone2} 2020-11-09T18:15:45.917 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T18:15:45.962 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T18:15:46.025 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ymzeui-40 zones={test-zone1, test-zone2} 2020-11-09T18:15:46.026 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ymzeui-40 zones={test-zone3, test-zone2} 2020-11-09T18:15:46.035 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T18:15:46.082 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T18:15:46.149 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ymzeui-40 zones={test-zone3, test-zone2} 2020-11-09T18:15:46.150 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ymzeui-40 zones={test-zone1, test-zone3} 2020-11-09T18:15:46.162 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T18:15:46.210 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T18:15:46.269 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ymzeui-40 zones={test-zone1, test-zone3} 2020-11-09T18:15:46.269 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ymzeui-40 zones={test-zone2, test-zone3} 2020-11-09T18:15:46.280 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T18:15:46.326 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T18:15:46.390 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ymzeui-40 zones={test-zone2, test-zone3} 2020-11-09T18:15:46.390 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ymzeui-41 zones={test-zone2, test-zone1} 2020-11-09T18:15:46.407 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T18:15:46.454 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T18:15:46.520 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ymzeui-41 zones={test-zone2, test-zone1} 2020-11-09T18:15:46.521 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ymzeui-41 zones={test-zone3, test-zone1} 2020-11-09T18:15:46.531 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T18:15:46.578 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T18:15:46.640 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ymzeui-41 zones={test-zone3, test-zone1} 2020-11-09T18:15:46.641 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ymzeui-41 zones={test-zone1, test-zone2} 2020-11-09T18:15:46.652 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T18:15:46.702 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T18:15:46.769 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ymzeui-41 zones={test-zone1, test-zone2} 2020-11-09T18:15:46.769 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ymzeui-41 zones={test-zone3, test-zone2} 2020-11-09T18:15:46.780 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T18:15:46.830 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T18:15:46.897 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ymzeui-41 zones={test-zone3, test-zone2} 2020-11-09T18:15:46.898 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ymzeui-41 zones={test-zone1, test-zone3} 2020-11-09T18:15:46.911 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T18:15:46.958 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T18:15:47.020 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ymzeui-41 zones={test-zone1, test-zone3} 2020-11-09T18:15:47.021 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ymzeui-41 zones={test-zone2, test-zone3} 2020-11-09T18:15:47.034 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T18:15:47.083 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T18:15:47.146 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ymzeui-41 zones={test-zone2, test-zone3} 2020-11-09T18:15:47.147 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ymzeui-42 zones={test-zone2, test-zone1} 2020-11-09T18:15:47.156 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T18:15:47.206 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T18:15:47.268 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ymzeui-42 zones={test-zone2, test-zone1} 2020-11-09T18:15:47.269 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ymzeui-42 zones={test-zone3, test-zone1} 2020-11-09T18:15:47.279 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T18:15:47.330 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T18:15:47.393 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ymzeui-42 zones={test-zone3, test-zone1} 2020-11-09T18:15:47.393 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ymzeui-42 zones={test-zone1, test-zone2} 2020-11-09T18:15:47.405 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T18:15:47.455 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T18:15:47.517 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ymzeui-42 zones={test-zone1, test-zone2} 2020-11-09T18:15:47.518 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ymzeui-42 zones={test-zone3, test-zone2} 2020-11-09T18:15:47.528 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T18:15:47.578 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T18:15:47.647 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ymzeui-42 zones={test-zone3, test-zone2} 2020-11-09T18:15:47.648 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ymzeui-42 zones={test-zone1, test-zone3} 2020-11-09T18:15:47.660 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T18:15:47.710 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T18:15:47.778 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ymzeui-42 zones={test-zone1, test-zone3} 2020-11-09T18:15:47.779 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ymzeui-42 zones={test-zone2, test-zone3} 2020-11-09T18:15:47.790 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T18:15:47.842 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T18:15:47.909 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ymzeui-42 zones={test-zone2, test-zone3} 2020-11-09T18:15:47.910 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ymzeui-43 zones={test-zone2, test-zone1} 2020-11-09T18:15:47.920 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T18:15:47.970 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T18:15:48.033 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ymzeui-43 zones={test-zone2, test-zone1} 2020-11-09T18:15:48.033 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ymzeui-43 zones={test-zone3, test-zone1} 2020-11-09T18:15:48.042 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T18:15:48.090 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T18:15:48.152 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ymzeui-43 zones={test-zone3, test-zone1} 2020-11-09T18:15:48.153 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ymzeui-43 zones={test-zone1, test-zone2} 2020-11-09T18:15:48.163 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T18:15:48.210 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T18:15:48.273 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ymzeui-43 zones={test-zone1, test-zone2} 2020-11-09T18:15:48.274 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ymzeui-43 zones={test-zone3, test-zone2} 2020-11-09T18:15:48.283 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T18:15:48.330 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T18:15:48.397 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ymzeui-43 zones={test-zone3, test-zone2} 2020-11-09T18:15:48.397 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ymzeui-43 zones={test-zone1, test-zone3} 2020-11-09T18:15:48.408 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T18:15:48.454 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T18:15:48.517 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ymzeui-43 zones={test-zone1, test-zone3} 2020-11-09T18:15:48.517 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ymzeui-43 zones={test-zone2, test-zone3} 2020-11-09T18:15:48.528 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T18:15:48.574 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T18:15:48.637 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ymzeui-43 zones={test-zone2, test-zone3} 2020-11-09T18:15:48.637 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ymzeui-44 zones={test-zone2, test-zone1} 2020-11-09T18:15:48.648 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T18:15:48.698 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T18:15:48.765 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ymzeui-44 zones={test-zone2, test-zone1} 2020-11-09T18:15:48.766 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ymzeui-44 zones={test-zone3, test-zone1} 2020-11-09T18:15:48.776 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T18:15:48.822 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T18:15:48.885 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ymzeui-44 zones={test-zone3, test-zone1} 2020-11-09T18:15:48.886 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ymzeui-44 zones={test-zone1, test-zone2} 2020-11-09T18:15:48.897 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T18:15:48.942 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T18:15:49.004 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ymzeui-44 zones={test-zone1, test-zone2} 2020-11-09T18:15:49.005 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ymzeui-44 zones={test-zone3, test-zone2} 2020-11-09T18:15:49.015 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T18:15:49.066 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T18:15:49.133 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ymzeui-44 zones={test-zone3, test-zone2} 2020-11-09T18:15:49.134 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ymzeui-44 zones={test-zone1, test-zone3} 2020-11-09T18:15:49.146 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T18:15:49.194 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T18:15:49.252 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ymzeui-44 zones={test-zone1, test-zone3} 2020-11-09T18:15:49.253 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ymzeui-44 zones={test-zone2, test-zone3} 2020-11-09T18:15:49.264 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T18:15:49.318 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T18:15:49.381 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ymzeui-44 zones={test-zone2, test-zone3} 2020-11-09T18:15:49.382 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ymzeui-45 zones={test-zone2, test-zone1} 2020-11-09T18:15:49.392 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T18:15:49.442 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T18:15:49.509 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ymzeui-45 zones={test-zone2, test-zone1} 2020-11-09T18:15:49.509 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ymzeui-45 zones={test-zone3, test-zone1} 2020-11-09T18:15:49.519 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T18:15:49.570 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T18:15:49.632 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ymzeui-45 zones={test-zone3, test-zone1} 2020-11-09T18:15:49.633 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ymzeui-45 zones={test-zone1, test-zone2} 2020-11-09T18:15:49.642 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T18:15:49.690 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T18:15:49.758 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ymzeui-45 zones={test-zone1, test-zone2} 2020-11-09T18:15:49.758 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ymzeui-45 zones={test-zone3, test-zone2} 2020-11-09T18:15:49.768 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T18:15:49.814 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T18:15:49.877 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ymzeui-45 zones={test-zone3, test-zone2} 2020-11-09T18:15:49.877 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ymzeui-45 zones={test-zone1, test-zone3} 2020-11-09T18:15:49.890 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T18:15:49.942 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T18:15:50.000 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ymzeui-45 zones={test-zone1, test-zone3} 2020-11-09T18:15:50.001 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ymzeui-45 zones={test-zone2, test-zone3} 2020-11-09T18:15:50.012 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T18:15:50.058 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T18:15:50.121 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ymzeui-45 zones={test-zone2, test-zone3} 2020-11-09T18:15:50.121 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:50.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:50.122 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:15:50.158 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:15:50.159 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:15:50.392 INFO:teuthology.orchestra.run.smithi051.stdout:[ 2020-11-09T18:15:50.392 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.392 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945439.484659_48.1", 2020-11-09T18:15:50.393 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "user", 2020-11-09T18:15:50.393 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "rgw-multisite-test-user", 2020-11-09T18:15:50.393 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:10:39.484659067Z", 2020-11-09T18:15:50.394 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.394 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.394 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.395 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.395 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.395 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.395 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_qxUmJfyn2gLxVOOHvE_t0VQ", 2020-11-09T18:15:50.396 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.396 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.396 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.396 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.397 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.397 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.397 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.397 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.398 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945439.492888_50.1", 2020-11-09T18:15:50.398 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "user", 2020-11-09T18:15:50.398 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "rgw-multisite-test-user", 2020-11-09T18:15:50.399 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:10:39.492888346Z", 2020-11-09T18:15:50.399 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.399 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.399 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.400 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.400 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.400 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.400 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_qxUmJfyn2gLxVOOHvE_t0VQ", 2020-11-09T18:15:50.401 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.401 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.401 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.402 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.402 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.402 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.402 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.403 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.403 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945586.256023_51.1", 2020-11-09T18:15:50.403 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.404 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-10:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.10", 2020-11-09T18:15:50.404 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:06.256023190Z", 2020-11-09T18:15:50.404 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.404 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.405 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.405 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.405 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.405 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.406 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_xk1WELKUHZIqmUd5oTGWrEo", 2020-11-09T18:15:50.406 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.406 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.406 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.406 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.407 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.407 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.407 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.407 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.407 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945586.261184_59.1", 2020-11-09T18:15:50.407 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.408 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-10:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.10", 2020-11-09T18:15:50.408 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:06.261184780Z", 2020-11-09T18:15:50.408 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.408 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.408 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.409 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.409 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.409 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.409 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_xk1WELKUHZIqmUd5oTGWrEo", 2020-11-09T18:15:50.409 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.410 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.410 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.410 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.410 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.410 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.410 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.411 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.411 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945586.263844_60.1", 2020-11-09T18:15:50.411 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.411 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-10", 2020-11-09T18:15:50.411 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:06.263844660Z", 2020-11-09T18:15:50.411 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.411 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.411 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.412 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.412 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.412 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.412 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_p-7-bX-sO2hh9J_2Nonu8rp", 2020-11-09T18:15:50.412 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.412 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.412 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.413 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.413 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.413 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.413 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.413 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.413 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945586.268581_61.1", 2020-11-09T18:15:50.414 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.414 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-10", 2020-11-09T18:15:50.414 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:06.268581555Z", 2020-11-09T18:15:50.414 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.414 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.415 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.415 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.415 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.415 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.416 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_p-7-bX-sO2hh9J_2Nonu8rp", 2020-11-09T18:15:50.416 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.416 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.416 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.416 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.417 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.417 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.417 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.417 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.417 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945587.202762_62.1", 2020-11-09T18:15:50.417 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.417 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-11:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.11", 2020-11-09T18:15:50.417 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:07.202762446Z", 2020-11-09T18:15:50.418 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.418 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.418 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.418 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.418 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.418 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.419 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_5opGpz-RxCA9co32z-sLP0e", 2020-11-09T18:15:50.419 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.419 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.419 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.419 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.419 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.419 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.420 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.420 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.420 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945587.207905_63.1", 2020-11-09T18:15:50.420 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.421 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-11:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.11", 2020-11-09T18:15:50.421 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:07.207905728Z", 2020-11-09T18:15:50.421 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.421 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.422 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.422 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.422 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.422 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.423 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_5opGpz-RxCA9co32z-sLP0e", 2020-11-09T18:15:50.423 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.423 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.423 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.423 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.423 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.424 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.424 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.424 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.424 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945587.210891_64.1", 2020-11-09T18:15:50.424 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.425 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-11", 2020-11-09T18:15:50.425 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:07.210891570Z", 2020-11-09T18:15:50.425 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.425 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.425 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.425 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.426 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.426 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.426 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_QiNkPq8a5F1hoeuAtqCxHNQ", 2020-11-09T18:15:50.426 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.427 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.427 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.427 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.427 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.428 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.428 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.428 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.428 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945587.216871_65.1", 2020-11-09T18:15:50.429 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.429 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-11", 2020-11-09T18:15:50.429 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:07.216871948Z", 2020-11-09T18:15:50.430 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.430 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.430 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.430 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.431 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.431 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.431 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_QiNkPq8a5F1hoeuAtqCxHNQ", 2020-11-09T18:15:50.431 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.431 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.432 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.432 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.432 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.432 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.433 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.433 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.433 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945588.126275_66.1", 2020-11-09T18:15:50.433 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.434 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-12:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.12", 2020-11-09T18:15:50.434 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:08.126275141Z", 2020-11-09T18:15:50.434 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.434 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.434 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.434 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.435 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.435 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.435 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_7WQjdW6VYC3fP3T1DUBlsCk", 2020-11-09T18:15:50.435 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.435 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.435 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.436 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.436 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.436 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.436 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.436 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.436 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945588.133515_67.1", 2020-11-09T18:15:50.436 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.436 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-12:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.12", 2020-11-09T18:15:50.437 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:08.133515560Z", 2020-11-09T18:15:50.437 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.437 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.437 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.437 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.437 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.437 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.438 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_7WQjdW6VYC3fP3T1DUBlsCk", 2020-11-09T18:15:50.438 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.438 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.438 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.438 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.438 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.439 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.439 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.439 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.439 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945588.135642_68.1", 2020-11-09T18:15:50.439 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.439 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-12", 2020-11-09T18:15:50.440 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:08.135642294Z", 2020-11-09T18:15:50.440 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.440 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.440 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.440 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.440 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.441 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.441 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_PdQ7asc_0k_3_XotzK7eatQ", 2020-11-09T18:15:50.441 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.441 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.441 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.442 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.442 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.442 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.442 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.442 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.442 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945588.143746_69.1", 2020-11-09T18:15:50.442 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.443 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-12", 2020-11-09T18:15:50.443 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:08.143746572Z", 2020-11-09T18:15:50.443 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.443 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.443 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.443 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.443 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.444 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.444 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_PdQ7asc_0k_3_XotzK7eatQ", 2020-11-09T18:15:50.444 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.444 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.444 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.445 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.445 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.445 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.445 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.445 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.445 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945588.381357_70.1", 2020-11-09T18:15:50.445 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.446 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-13:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.13", 2020-11-09T18:15:50.446 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:08.381357113Z", 2020-11-09T18:15:50.446 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.446 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.446 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.446 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.446 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.447 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.447 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_jRsLDHCkxygm_NRmusaemxh", 2020-11-09T18:15:50.447 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.447 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.447 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.447 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.447 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.448 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.448 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.448 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.448 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945588.386292_71.1", 2020-11-09T18:15:50.448 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.449 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-13:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.13", 2020-11-09T18:15:50.449 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:08.386292596Z", 2020-11-09T18:15:50.449 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.449 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.449 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.449 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.449 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.449 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.450 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_jRsLDHCkxygm_NRmusaemxh", 2020-11-09T18:15:50.450 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.450 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.450 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.450 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.450 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.451 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.451 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.451 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.451 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945588.391790_72.1", 2020-11-09T18:15:50.452 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.452 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-13", 2020-11-09T18:15:50.452 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:08.391790067Z", 2020-11-09T18:15:50.452 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.452 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.452 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.452 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.453 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.453 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.453 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_1zAxS7Eu4GCsgwIw7eCrrnz", 2020-11-09T18:15:50.453 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.453 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.453 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.453 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.454 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.454 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.454 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.454 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.454 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945588.397490_73.1", 2020-11-09T18:15:50.454 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.454 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-13", 2020-11-09T18:15:50.455 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:08.397490757Z", 2020-11-09T18:15:50.455 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.455 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.455 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.455 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.455 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.456 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.456 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_1zAxS7Eu4GCsgwIw7eCrrnz", 2020-11-09T18:15:50.456 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.456 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.456 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.456 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.457 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.457 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.457 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.457 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.457 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945589.380394_74.1", 2020-11-09T18:15:50.457 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.458 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-14:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.14", 2020-11-09T18:15:50.458 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:09.380394647Z", 2020-11-09T18:15:50.458 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.458 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.458 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.458 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.459 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.459 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.459 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_ZoR7I5lOnVtZkMZ3aHO24Vp", 2020-11-09T18:15:50.459 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.459 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.459 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.460 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.460 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.460 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.460 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.460 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.460 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945589.385117_75.1", 2020-11-09T18:15:50.460 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.461 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-14:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.14", 2020-11-09T18:15:50.461 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:09.385117237Z", 2020-11-09T18:15:50.461 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.461 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.461 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.462 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.462 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.462 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.462 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_ZoR7I5lOnVtZkMZ3aHO24Vp", 2020-11-09T18:15:50.462 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.462 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.463 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.463 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.463 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.463 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.463 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.465 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.465 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945589.387155_76.1", 2020-11-09T18:15:50.465 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.466 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-14", 2020-11-09T18:15:50.466 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:09.387155581Z", 2020-11-09T18:15:50.466 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.466 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.466 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.466 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.466 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.467 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.467 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_iQYh1hCLyur_paDmK5MzSiw", 2020-11-09T18:15:50.467 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.467 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.467 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.467 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.467 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.468 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.468 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.468 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.468 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945589.391251_77.1", 2020-11-09T18:15:50.468 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.468 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-14", 2020-11-09T18:15:50.469 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:09.391251540Z", 2020-11-09T18:15:50.469 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.469 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.469 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.469 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.469 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.469 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.470 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_iQYh1hCLyur_paDmK5MzSiw", 2020-11-09T18:15:50.470 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.470 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.470 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.470 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.471 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.471 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.471 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.471 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.471 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945589.523751_78.1", 2020-11-09T18:15:50.471 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.472 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-15:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.15", 2020-11-09T18:15:50.472 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:09.523751440Z", 2020-11-09T18:15:50.472 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.472 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.473 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.473 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.473 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.473 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.473 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_KVNlDu4zRBEGnM1a31uWniG", 2020-11-09T18:15:50.473 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.474 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.474 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.475 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.475 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.475 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.475 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.475 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.475 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945589.534160_79.1", 2020-11-09T18:15:50.475 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.476 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-15:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.15", 2020-11-09T18:15:50.476 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:09.534160171Z", 2020-11-09T18:15:50.476 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.476 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.476 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.476 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.477 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.477 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.477 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_KVNlDu4zRBEGnM1a31uWniG", 2020-11-09T18:15:50.477 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.477 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.477 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.477 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.478 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.478 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.478 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.478 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.478 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945589.536862_80.1", 2020-11-09T18:15:50.478 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.478 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-15", 2020-11-09T18:15:50.479 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:09.536862933Z", 2020-11-09T18:15:50.479 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.479 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.479 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.479 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.479 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.479 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.480 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_Cq6C5mA4ASiylyTIC5SMBlP", 2020-11-09T18:15:50.480 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.480 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.480 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.480 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.480 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.480 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.480 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.481 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.481 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945589.541920_81.1", 2020-11-09T18:15:50.481 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.481 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-15", 2020-11-09T18:15:50.481 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:09.541920734Z", 2020-11-09T18:15:50.481 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.481 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.482 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.483 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.483 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.483 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.484 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_Cq6C5mA4ASiylyTIC5SMBlP", 2020-11-09T18:15:50.484 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.484 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.484 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.485 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.485 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.485 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.485 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.485 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.485 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945590.011767_82.1", 2020-11-09T18:15:50.486 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.486 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-16:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.16", 2020-11-09T18:15:50.486 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:10.011767329Z", 2020-11-09T18:15:50.486 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.486 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.486 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.486 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.487 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.487 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.487 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_fJsd-7NZirDCEodHfdpY9oA", 2020-11-09T18:15:50.487 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.487 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.487 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.487 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.488 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.488 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.488 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.488 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.488 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945590.016722_83.1", 2020-11-09T18:15:50.488 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.489 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-16:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.16", 2020-11-09T18:15:50.490 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:10.016722284Z", 2020-11-09T18:15:50.490 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.490 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.490 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.490 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.490 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.490 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.491 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_fJsd-7NZirDCEodHfdpY9oA", 2020-11-09T18:15:50.491 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.491 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.491 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.491 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.491 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.492 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.492 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.492 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.492 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945590.019556_84.1", 2020-11-09T18:15:50.492 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.492 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-16", 2020-11-09T18:15:50.492 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:10.019556366Z", 2020-11-09T18:15:50.493 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.493 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.493 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.493 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.493 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.494 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.494 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_83VseKNDEzwg2zXJyAxVSNV", 2020-11-09T18:15:50.494 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.494 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.494 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.494 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.494 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.495 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.495 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.495 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.495 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945590.028620_85.1", 2020-11-09T18:15:50.495 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.495 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-16", 2020-11-09T18:15:50.495 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:10.028620030Z", 2020-11-09T18:15:50.495 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.496 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.496 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.496 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.496 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.496 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.496 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_83VseKNDEzwg2zXJyAxVSNV", 2020-11-09T18:15:50.496 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.497 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.497 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.497 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.497 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.497 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.497 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.497 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.497 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945590.059189_86.1", 2020-11-09T18:15:50.498 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.498 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-17:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.17", 2020-11-09T18:15:50.498 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:10.059189312Z", 2020-11-09T18:15:50.498 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.498 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.498 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.498 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.499 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.499 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.499 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_nhpbrE0BaTecbpBGZZxkC_9", 2020-11-09T18:15:50.499 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.499 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.501 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.502 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.502 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.502 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.502 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.503 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.503 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945590.065317_87.1", 2020-11-09T18:15:50.503 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.503 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-17:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.17", 2020-11-09T18:15:50.503 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:10.065317701Z", 2020-11-09T18:15:50.503 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.503 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.504 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.504 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.504 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.504 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.504 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_nhpbrE0BaTecbpBGZZxkC_9", 2020-11-09T18:15:50.504 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.504 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.505 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.505 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.505 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.505 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.505 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.505 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.505 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945590.068336_88.1", 2020-11-09T18:15:50.506 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.506 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-17", 2020-11-09T18:15:50.506 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:10.068336838Z", 2020-11-09T18:15:50.506 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.506 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.507 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.507 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.508 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.508 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.508 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_DZ4ozYh7iBiXpSqfO2MFhvn", 2020-11-09T18:15:50.508 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.509 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.509 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.509 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.509 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.509 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.510 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.510 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.510 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945590.076749_89.1", 2020-11-09T18:15:50.510 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.510 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-17", 2020-11-09T18:15:50.510 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:10.076749917Z", 2020-11-09T18:15:50.510 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.511 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.511 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.511 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.511 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.511 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.511 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_DZ4ozYh7iBiXpSqfO2MFhvn", 2020-11-09T18:15:50.512 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.512 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.512 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.512 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.513 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.513 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.513 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.514 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.514 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945590.120726_90.1", 2020-11-09T18:15:50.514 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.514 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-18:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.18", 2020-11-09T18:15:50.515 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:10.120726234Z", 2020-11-09T18:15:50.515 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.515 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.515 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.515 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.516 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.516 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.516 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_R8b8xLVdGNeTGDdC42EuEfE", 2020-11-09T18:15:50.516 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.516 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.516 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.516 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.517 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.517 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.517 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.517 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.517 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945590.127213_91.1", 2020-11-09T18:15:50.517 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.517 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-18:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.18", 2020-11-09T18:15:50.518 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:10.127213093Z", 2020-11-09T18:15:50.518 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.518 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.518 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.518 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.519 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.519 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.519 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_R8b8xLVdGNeTGDdC42EuEfE", 2020-11-09T18:15:50.519 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.519 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.519 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.519 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.520 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.520 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.520 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.520 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.520 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945590.129116_92.1", 2020-11-09T18:15:50.520 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.520 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-18", 2020-11-09T18:15:50.521 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:10.129116612Z", 2020-11-09T18:15:50.521 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.521 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.521 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.521 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.521 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.521 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.521 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_jvrbBsWxGWFBmAxvo-GL8oI", 2020-11-09T18:15:50.522 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.522 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.522 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.522 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.522 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.522 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.522 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.523 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.523 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945590.133065_93.1", 2020-11-09T18:15:50.523 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.523 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-18", 2020-11-09T18:15:50.523 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:10.133065786Z", 2020-11-09T18:15:50.524 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.524 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.524 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.524 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.525 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.525 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.525 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_jvrbBsWxGWFBmAxvo-GL8oI", 2020-11-09T18:15:50.526 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.526 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.526 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.527 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.527 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.527 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.528 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.528 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.528 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945590.156864_94.1", 2020-11-09T18:15:50.528 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.529 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-16:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.16", 2020-11-09T18:15:50.529 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:10.156864142Z", 2020-11-09T18:15:50.529 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.530 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.530 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_fJsd-7NZirDCEodHfdpY9oA", 2020-11-09T18:15:50.530 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.531 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.531 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.531 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_fJsd-7NZirDCEodHfdpY9oA", 2020-11-09T18:15:50.531 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:50.532 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.532 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.532 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.533 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.533 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.533 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.533 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.534 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945590.163596_95.1", 2020-11-09T18:15:50.534 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.534 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-16:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.16", 2020-11-09T18:15:50.534 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:10.163596031Z", 2020-11-09T18:15:50.535 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.535 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.535 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_fJsd-7NZirDCEodHfdpY9oA", 2020-11-09T18:15:50.535 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.536 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.536 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.536 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_fJsd-7NZirDCEodHfdpY9oA", 2020-11-09T18:15:50.537 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:50.537 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.537 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.537 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.538 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.538 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.538 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.538 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.538 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945590.168397_96.1", 2020-11-09T18:15:50.539 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.539 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-17:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.17", 2020-11-09T18:15:50.539 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:10.168397876Z", 2020-11-09T18:15:50.539 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.539 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.540 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_nhpbrE0BaTecbpBGZZxkC_9", 2020-11-09T18:15:50.540 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.540 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.540 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.540 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_nhpbrE0BaTecbpBGZZxkC_9", 2020-11-09T18:15:50.540 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:50.541 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.541 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.541 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.541 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.541 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.541 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.541 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.541 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945590.178873_97.1", 2020-11-09T18:15:50.542 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.542 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-17:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.17", 2020-11-09T18:15:50.542 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:10.178873937Z", 2020-11-09T18:15:50.542 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.542 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.542 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_nhpbrE0BaTecbpBGZZxkC_9", 2020-11-09T18:15:50.543 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.543 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.543 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.543 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_nhpbrE0BaTecbpBGZZxkC_9", 2020-11-09T18:15:50.543 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:50.543 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.543 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.544 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.544 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.544 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.544 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.544 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.544 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945590.190493_98.1", 2020-11-09T18:15:50.544 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.544 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-18:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.18", 2020-11-09T18:15:50.545 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:10.190493784Z", 2020-11-09T18:15:50.545 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.545 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.545 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_R8b8xLVdGNeTGDdC42EuEfE", 2020-11-09T18:15:50.545 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.545 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.545 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.546 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_R8b8xLVdGNeTGDdC42EuEfE", 2020-11-09T18:15:50.546 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:50.546 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.546 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.546 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.546 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.546 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.547 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.547 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.547 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945590.199376_99.1", 2020-11-09T18:15:50.547 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.547 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-18:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.18", 2020-11-09T18:15:50.547 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:10.199376961Z", 2020-11-09T18:15:50.547 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.547 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.548 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_R8b8xLVdGNeTGDdC42EuEfE", 2020-11-09T18:15:50.548 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.548 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.548 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.548 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_R8b8xLVdGNeTGDdC42EuEfE", 2020-11-09T18:15:50.548 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:50.549 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.549 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.549 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.549 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.549 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.550 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.550 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.550 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945592.270499_100.1", 2020-11-09T18:15:50.550 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.550 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-19:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.19", 2020-11-09T18:15:50.550 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:12.270499729Z", 2020-11-09T18:15:50.550 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.551 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.551 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.551 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.551 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.551 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.551 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_E6pXupKMFQLC75ImCoCkV74", 2020-11-09T18:15:50.551 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.551 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.552 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.552 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.552 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.552 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.552 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.552 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.552 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945592.275423_102.1", 2020-11-09T18:15:50.553 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.553 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-19:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.19", 2020-11-09T18:15:50.553 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:12.275423973Z", 2020-11-09T18:15:50.553 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.553 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.553 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.553 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.554 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.554 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.554 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_E6pXupKMFQLC75ImCoCkV74", 2020-11-09T18:15:50.554 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.554 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.554 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.554 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.555 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.555 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.555 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.555 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.555 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945592.278527_103.1", 2020-11-09T18:15:50.555 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.555 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-19", 2020-11-09T18:15:50.556 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:12.278527677Z", 2020-11-09T18:15:50.556 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.556 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.556 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.556 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.556 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.556 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.556 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_OGzY_hYHGuuTc6_1SixJvo6", 2020-11-09T18:15:50.557 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.557 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.557 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.557 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.557 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.557 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.557 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.558 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.558 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945592.284263_104.1", 2020-11-09T18:15:50.558 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.558 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-19", 2020-11-09T18:15:50.558 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:12.284263022Z", 2020-11-09T18:15:50.558 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.558 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.559 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.559 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.559 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.559 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.559 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_OGzY_hYHGuuTc6_1SixJvo6", 2020-11-09T18:15:50.559 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.559 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.560 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.560 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.560 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.560 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.560 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.560 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.560 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945592.291372_105.1", 2020-11-09T18:15:50.561 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.561 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-19:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.19", 2020-11-09T18:15:50.561 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:12.291372646Z", 2020-11-09T18:15:50.561 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.561 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.561 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_E6pXupKMFQLC75ImCoCkV74", 2020-11-09T18:15:50.561 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.561 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.562 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.562 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_E6pXupKMFQLC75ImCoCkV74", 2020-11-09T18:15:50.562 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:50.562 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.562 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.562 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.562 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.563 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.563 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.563 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.563 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945592.300193_106.1", 2020-11-09T18:15:50.563 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.563 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-19:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.19", 2020-11-09T18:15:50.563 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:12.300193728Z", 2020-11-09T18:15:50.564 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.564 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.564 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_E6pXupKMFQLC75ImCoCkV74", 2020-11-09T18:15:50.564 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.564 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.564 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.564 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_E6pXupKMFQLC75ImCoCkV74", 2020-11-09T18:15:50.565 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:50.565 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.565 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.565 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.565 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.565 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.565 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.565 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.566 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945594.039798_107.1", 2020-11-09T18:15:50.566 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.566 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-20:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.20", 2020-11-09T18:15:50.566 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:14.039798004Z", 2020-11-09T18:15:50.566 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.566 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.566 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.567 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.567 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.567 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.567 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_cfJ_IuSbXauFncnrlMQUrK1", 2020-11-09T18:15:50.567 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.567 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.567 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.568 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.568 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.568 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.568 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.568 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.568 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945594.043998_109.1", 2020-11-09T18:15:50.568 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.569 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-20:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.20", 2020-11-09T18:15:50.569 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:14.043998666Z", 2020-11-09T18:15:50.569 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.569 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.569 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.569 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.569 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.570 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.570 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_cfJ_IuSbXauFncnrlMQUrK1", 2020-11-09T18:15:50.570 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.570 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.570 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.570 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.570 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.571 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.571 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.571 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.571 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945594.046065_110.1", 2020-11-09T18:15:50.571 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.571 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-20", 2020-11-09T18:15:50.571 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:14.046065820Z", 2020-11-09T18:15:50.572 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.572 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.572 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.572 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.572 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.572 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.573 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_4-ey3kn91fmGef8EsodcNMf", 2020-11-09T18:15:50.573 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.573 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.573 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.573 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.573 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.573 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.574 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.574 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.574 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945594.050612_111.1", 2020-11-09T18:15:50.574 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.574 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-20", 2020-11-09T18:15:50.574 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:14.050612141Z", 2020-11-09T18:15:50.575 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.575 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.575 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.575 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.575 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.575 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.575 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_4-ey3kn91fmGef8EsodcNMf", 2020-11-09T18:15:50.576 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.576 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.576 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.576 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.576 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.576 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.577 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.577 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.577 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945595.459954_112.1", 2020-11-09T18:15:50.577 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.577 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-21:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.21", 2020-11-09T18:15:50.577 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:15.459954913Z", 2020-11-09T18:15:50.577 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.578 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.578 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.578 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.578 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.578 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.578 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_7weq2-_FEkT51PfhS1TopF2", 2020-11-09T18:15:50.579 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.579 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.579 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.579 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.579 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.579 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.579 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.579 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.580 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945595.464408_114.1", 2020-11-09T18:15:50.580 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.580 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-21:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.21", 2020-11-09T18:15:50.580 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:15.464408755Z", 2020-11-09T18:15:50.580 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.580 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.580 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.581 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.581 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.581 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.581 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_7weq2-_FEkT51PfhS1TopF2", 2020-11-09T18:15:50.581 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.581 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.581 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.582 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.582 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.582 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.582 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.582 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.582 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945595.466475_115.1", 2020-11-09T18:15:50.582 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.583 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-21", 2020-11-09T18:15:50.583 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:15.466475089Z", 2020-11-09T18:15:50.583 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.583 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.583 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.583 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.583 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.584 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.584 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_cxcDXnujWtafpwp1_Zf7_7E", 2020-11-09T18:15:50.584 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.584 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.584 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.585 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.585 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.585 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.585 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.585 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.585 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945595.470860_116.1", 2020-11-09T18:15:50.585 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.586 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-21", 2020-11-09T18:15:50.586 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:15.470860202Z", 2020-11-09T18:15:50.586 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.586 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.586 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.586 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.587 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.587 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.587 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_cxcDXnujWtafpwp1_Zf7_7E", 2020-11-09T18:15:50.587 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.587 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.587 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.587 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.588 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.588 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.588 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.588 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.588 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945595.574082_117.1", 2020-11-09T18:15:50.588 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.589 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-22:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.22", 2020-11-09T18:15:50.589 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:15.574082160Z", 2020-11-09T18:15:50.589 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.589 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.589 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.589 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.589 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.590 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.590 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_gu325NeoTQ-n6-N4RZO_Ufq", 2020-11-09T18:15:50.590 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.590 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.590 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.590 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.591 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.591 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.591 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.591 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.591 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945595.578263_118.1", 2020-11-09T18:15:50.591 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.591 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-22:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.22", 2020-11-09T18:15:50.592 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:15.578263069Z", 2020-11-09T18:15:50.592 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.592 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.592 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.592 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.592 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.593 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.593 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_gu325NeoTQ-n6-N4RZO_Ufq", 2020-11-09T18:15:50.593 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.593 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.593 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.593 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.593 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.594 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.594 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.594 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.594 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945595.580706_119.1", 2020-11-09T18:15:50.594 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.594 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-22", 2020-11-09T18:15:50.594 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:15.580706536Z", 2020-11-09T18:15:50.595 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.595 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.595 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.595 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.595 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.595 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.595 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_lHWYslSbF-coDhm_0XUYDqP", 2020-11-09T18:15:50.595 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.596 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.596 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.596 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.596 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.596 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.596 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.597 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.597 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945595.585376_120.1", 2020-11-09T18:15:50.597 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.597 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-22", 2020-11-09T18:15:50.597 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:15.585376092Z", 2020-11-09T18:15:50.597 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.597 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.598 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.598 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.598 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.598 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.598 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_lHWYslSbF-coDhm_0XUYDqP", 2020-11-09T18:15:50.598 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.598 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.599 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.599 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.599 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.599 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.599 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.599 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.600 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945595.651287_121.1", 2020-11-09T18:15:50.600 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.600 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-23:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.23", 2020-11-09T18:15:50.600 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:15.651287758Z", 2020-11-09T18:15:50.600 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.600 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.600 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.601 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.601 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.601 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.601 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_k8tGHuX9HMmJcr5VgC0V0M4", 2020-11-09T18:15:50.601 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.601 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.601 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.602 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.602 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.602 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.602 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.602 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.602 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945595.655554_122.1", 2020-11-09T18:15:50.602 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.603 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-23:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.23", 2020-11-09T18:15:50.603 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:15.655554732Z", 2020-11-09T18:15:50.603 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.603 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.603 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.603 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.603 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.604 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.604 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_k8tGHuX9HMmJcr5VgC0V0M4", 2020-11-09T18:15:50.604 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.604 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.604 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.604 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.604 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.605 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.605 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.605 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.605 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945595.657606_123.1", 2020-11-09T18:15:50.605 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.606 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-23", 2020-11-09T18:15:50.606 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:15.657606349Z", 2020-11-09T18:15:50.606 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.606 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.606 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.606 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.607 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.607 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.607 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_rDxg2fTzKxNRTnXPuogbkec", 2020-11-09T18:15:50.607 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.607 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.607 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.607 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.608 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.608 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.608 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.608 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.608 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945595.662266_124.1", 2020-11-09T18:15:50.608 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.609 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-23", 2020-11-09T18:15:50.609 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:15.662266951Z", 2020-11-09T18:15:50.609 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.609 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.609 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.609 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.609 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.610 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.610 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_rDxg2fTzKxNRTnXPuogbkec", 2020-11-09T18:15:50.610 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.610 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.610 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.610 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.610 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.611 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.611 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.611 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.611 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945595.694579_125.1", 2020-11-09T18:15:50.611 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.611 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-21:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.21", 2020-11-09T18:15:50.611 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:15.694579560Z", 2020-11-09T18:15:50.612 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.612 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.612 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_7weq2-_FEkT51PfhS1TopF2", 2020-11-09T18:15:50.612 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.612 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.612 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.612 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_7weq2-_FEkT51PfhS1TopF2", 2020-11-09T18:15:50.612 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:50.613 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.613 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.613 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.613 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.613 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.613 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.613 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.614 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945595.699216_126.1", 2020-11-09T18:15:50.614 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.614 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-21:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.21", 2020-11-09T18:15:50.614 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:15.699216478Z", 2020-11-09T18:15:50.614 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.614 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.614 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_7weq2-_FEkT51PfhS1TopF2", 2020-11-09T18:15:50.614 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.615 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.615 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.615 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_7weq2-_FEkT51PfhS1TopF2", 2020-11-09T18:15:50.615 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:50.615 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.615 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.615 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.616 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.616 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.616 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.616 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.616 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945595.704101_127.1", 2020-11-09T18:15:50.616 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.616 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-22:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.22", 2020-11-09T18:15:50.617 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:15.704101323Z", 2020-11-09T18:15:50.617 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.617 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.617 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_gu325NeoTQ-n6-N4RZO_Ufq", 2020-11-09T18:15:50.617 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.617 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.617 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.617 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_gu325NeoTQ-n6-N4RZO_Ufq", 2020-11-09T18:15:50.618 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:50.618 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.618 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.618 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.618 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.618 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.618 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.619 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.619 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945595.709327_128.1", 2020-11-09T18:15:50.619 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.619 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-22:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.22", 2020-11-09T18:15:50.619 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:15.709327694Z", 2020-11-09T18:15:50.619 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.620 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.620 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_gu325NeoTQ-n6-N4RZO_Ufq", 2020-11-09T18:15:50.620 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.620 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.620 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.620 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_gu325NeoTQ-n6-N4RZO_Ufq", 2020-11-09T18:15:50.621 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:50.621 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.621 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.621 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.621 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.621 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.622 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.622 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.622 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945595.722945_129.1", 2020-11-09T18:15:50.622 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.622 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-23:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.23", 2020-11-09T18:15:50.622 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:15.722945874Z", 2020-11-09T18:15:50.623 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.623 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.623 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_k8tGHuX9HMmJcr5VgC0V0M4", 2020-11-09T18:15:50.623 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.623 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.623 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.623 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_k8tGHuX9HMmJcr5VgC0V0M4", 2020-11-09T18:15:50.624 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:50.624 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.624 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.624 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.624 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.624 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.624 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.625 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.625 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945595.729371_130.1", 2020-11-09T18:15:50.625 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.625 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-23:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.23", 2020-11-09T18:15:50.625 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:15.729371675Z", 2020-11-09T18:15:50.625 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.625 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.625 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_k8tGHuX9HMmJcr5VgC0V0M4", 2020-11-09T18:15:50.626 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.626 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.626 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.626 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_k8tGHuX9HMmJcr5VgC0V0M4", 2020-11-09T18:15:50.626 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:50.626 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.626 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.627 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.627 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.627 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.627 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.627 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.627 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945597.242843_131.1", 2020-11-09T18:15:50.627 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.628 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-24:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.24", 2020-11-09T18:15:50.628 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:17.242843380Z", 2020-11-09T18:15:50.628 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.628 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.628 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.628 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.628 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.629 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.629 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_ZWaAaUyrIgJWu47a8QLknCB", 2020-11-09T18:15:50.629 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.629 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.629 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.629 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.630 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.630 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.630 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.630 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.630 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945597.247361_133.1", 2020-11-09T18:15:50.630 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.631 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-24:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.24", 2020-11-09T18:15:50.631 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:17.247361813Z", 2020-11-09T18:15:50.631 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.631 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.631 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.631 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.631 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.632 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.632 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_ZWaAaUyrIgJWu47a8QLknCB", 2020-11-09T18:15:50.632 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.632 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.632 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.632 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.633 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.633 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.633 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.633 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.633 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945597.249391_134.1", 2020-11-09T18:15:50.633 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.633 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-24", 2020-11-09T18:15:50.634 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:17.249391359Z", 2020-11-09T18:15:50.634 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.634 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.634 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.634 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.634 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.635 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.635 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_M6yzvodjSwrGlU0XXU23tmw", 2020-11-09T18:15:50.635 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.635 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.635 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.635 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.635 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.636 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.636 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.636 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.636 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945597.257906_135.1", 2020-11-09T18:15:50.636 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.636 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-24", 2020-11-09T18:15:50.636 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:17.257906751Z", 2020-11-09T18:15:50.637 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.637 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.637 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.637 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.637 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.637 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.637 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_M6yzvodjSwrGlU0XXU23tmw", 2020-11-09T18:15:50.638 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.638 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.638 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.638 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.638 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.638 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.638 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.639 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.639 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945598.021538_136.1", 2020-11-09T18:15:50.639 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.639 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-25:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.25", 2020-11-09T18:15:50.639 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:18.021538398Z", 2020-11-09T18:15:50.639 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.639 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.639 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.640 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.640 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.640 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.640 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dm5hw1Gq9w9JdDaVBV0h96_", 2020-11-09T18:15:50.640 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.640 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.640 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.641 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.641 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.641 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.641 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.641 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.641 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945598.026407_137.1", 2020-11-09T18:15:50.641 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.642 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-25:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.25", 2020-11-09T18:15:50.642 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:18.026407438Z", 2020-11-09T18:15:50.642 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.642 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.642 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.642 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.642 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.643 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.643 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dm5hw1Gq9w9JdDaVBV0h96_", 2020-11-09T18:15:50.643 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.643 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.643 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.643 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.643 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.643 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.644 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.644 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.644 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945598.028359_138.1", 2020-11-09T18:15:50.644 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.644 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-25", 2020-11-09T18:15:50.644 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:18.028359151Z", 2020-11-09T18:15:50.645 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.645 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.645 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.645 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.645 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.645 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.646 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_hJWxzyq2VXkrZe0lTr6UJ0X", 2020-11-09T18:15:50.646 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.646 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.646 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.646 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.646 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.647 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.647 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.647 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.647 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945598.032307_139.1", 2020-11-09T18:15:50.647 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.647 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-25", 2020-11-09T18:15:50.647 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:18.032307526Z", 2020-11-09T18:15:50.648 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.648 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.648 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.648 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.648 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.648 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.649 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_hJWxzyq2VXkrZe0lTr6UJ0X", 2020-11-09T18:15:50.649 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.649 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.649 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.649 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.649 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.649 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.650 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.650 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.650 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945598.347441_140.1", 2020-11-09T18:15:50.650 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.650 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-26:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.26", 2020-11-09T18:15:50.650 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:18.347441566Z", 2020-11-09T18:15:50.650 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.651 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.651 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.651 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.651 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.651 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.651 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_tUH5PREzBpkbjt0vmvKAgOc", 2020-11-09T18:15:50.652 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.652 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.652 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.652 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.652 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.652 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.652 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.653 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.653 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945598.351465_141.1", 2020-11-09T18:15:50.653 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.653 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-26:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.26", 2020-11-09T18:15:50.653 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:18.351465217Z", 2020-11-09T18:15:50.653 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.653 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.654 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.654 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.654 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.654 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.654 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_tUH5PREzBpkbjt0vmvKAgOc", 2020-11-09T18:15:50.654 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.654 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.655 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.655 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.655 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.655 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.655 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.655 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.655 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945598.353325_142.1", 2020-11-09T18:15:50.655 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.656 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-26", 2020-11-09T18:15:50.656 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:18.353325221Z", 2020-11-09T18:15:50.656 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.656 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.656 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.656 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.656 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.657 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.657 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_Z6tqFl2xPji_I9drTzIP2KZ", 2020-11-09T18:15:50.657 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.657 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.657 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.657 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.657 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.658 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.658 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.658 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.658 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945598.357392_143.1", 2020-11-09T18:15:50.658 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.658 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-26", 2020-11-09T18:15:50.658 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:18.357392493Z", 2020-11-09T18:15:50.658 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.659 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.659 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.659 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.659 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.659 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.659 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_Z6tqFl2xPji_I9drTzIP2KZ", 2020-11-09T18:15:50.659 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.660 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.660 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.660 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.660 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.660 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.660 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.660 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.660 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945598.450979_144.1", 2020-11-09T18:15:50.661 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.661 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-24:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.24", 2020-11-09T18:15:50.661 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:18.450979415Z", 2020-11-09T18:15:50.661 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.661 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.661 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_ZWaAaUyrIgJWu47a8QLknCB", 2020-11-09T18:15:50.661 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.662 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.662 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.662 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_ZWaAaUyrIgJWu47a8QLknCB", 2020-11-09T18:15:50.662 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:50.662 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.662 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.663 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.663 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.663 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.663 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.663 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.663 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945598.461497_145.1", 2020-11-09T18:15:50.663 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.664 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-24:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.24", 2020-11-09T18:15:50.664 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:18.461497100Z", 2020-11-09T18:15:50.664 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.664 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.664 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_ZWaAaUyrIgJWu47a8QLknCB", 2020-11-09T18:15:50.664 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.665 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.665 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.665 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_ZWaAaUyrIgJWu47a8QLknCB", 2020-11-09T18:15:50.665 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:50.665 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.665 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.665 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.666 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.666 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.666 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.666 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.666 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945598.470055_146.1", 2020-11-09T18:15:50.666 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.667 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-24:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.24", 2020-11-09T18:15:50.667 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:18.470055992Z", 2020-11-09T18:15:50.667 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.667 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.667 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_ZWaAaUyrIgJWu47a8QLknCB", 2020-11-09T18:15:50.667 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:50.667 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.668 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.668 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_ZWaAaUyrIgJWu47a8QLknCB", 2020-11-09T18:15:50.668 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 3 2020-11-09T18:15:50.668 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.668 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.668 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.668 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.669 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.669 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.669 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.669 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945598.478206_147.1", 2020-11-09T18:15:50.669 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.669 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-24:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.24", 2020-11-09T18:15:50.669 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:18.478206910Z", 2020-11-09T18:15:50.669 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.670 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.670 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_ZWaAaUyrIgJWu47a8QLknCB", 2020-11-09T18:15:50.670 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:50.670 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.670 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.670 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_ZWaAaUyrIgJWu47a8QLknCB", 2020-11-09T18:15:50.670 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 3 2020-11-09T18:15:50.671 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.671 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.671 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.671 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.671 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.671 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.671 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.672 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945598.482582_148.1", 2020-11-09T18:15:50.672 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.672 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-25:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.25", 2020-11-09T18:15:50.672 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:18.482582905Z", 2020-11-09T18:15:50.672 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.673 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.673 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dm5hw1Gq9w9JdDaVBV0h96_", 2020-11-09T18:15:50.673 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.673 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.673 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.673 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dm5hw1Gq9w9JdDaVBV0h96_", 2020-11-09T18:15:50.673 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:50.674 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.674 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.674 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.674 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.674 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.674 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.675 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.675 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945598.488285_149.1", 2020-11-09T18:15:50.675 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.675 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-25:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.25", 2020-11-09T18:15:50.675 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:18.488285331Z", 2020-11-09T18:15:50.675 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.676 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.676 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dm5hw1Gq9w9JdDaVBV0h96_", 2020-11-09T18:15:50.676 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.676 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.676 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.676 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dm5hw1Gq9w9JdDaVBV0h96_", 2020-11-09T18:15:50.676 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:50.677 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.677 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.677 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.677 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.677 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.677 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.678 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.678 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945598.495708_150.1", 2020-11-09T18:15:50.678 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.678 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-25:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.25", 2020-11-09T18:15:50.678 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:18.495708983Z", 2020-11-09T18:15:50.678 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.678 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.679 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dm5hw1Gq9w9JdDaVBV0h96_", 2020-11-09T18:15:50.679 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:50.679 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.679 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.679 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dm5hw1Gq9w9JdDaVBV0h96_", 2020-11-09T18:15:50.679 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 3 2020-11-09T18:15:50.679 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.680 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.680 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.680 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.680 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.680 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.680 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.681 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945598.502185_151.1", 2020-11-09T18:15:50.681 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.681 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-25:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.25", 2020-11-09T18:15:50.681 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:18.502185684Z", 2020-11-09T18:15:50.681 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.681 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.681 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dm5hw1Gq9w9JdDaVBV0h96_", 2020-11-09T18:15:50.682 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:50.682 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.682 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.682 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dm5hw1Gq9w9JdDaVBV0h96_", 2020-11-09T18:15:50.682 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 3 2020-11-09T18:15:50.682 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.682 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.683 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.683 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.683 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.683 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.683 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.683 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945598.514968_152.1", 2020-11-09T18:15:50.683 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.683 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-26:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.26", 2020-11-09T18:15:50.684 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:18.514968069Z", 2020-11-09T18:15:50.684 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.684 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.684 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_tUH5PREzBpkbjt0vmvKAgOc", 2020-11-09T18:15:50.684 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.684 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.684 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.685 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_tUH5PREzBpkbjt0vmvKAgOc", 2020-11-09T18:15:50.685 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:50.685 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.685 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.685 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.685 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.685 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.685 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.686 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.686 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945598.528310_153.1", 2020-11-09T18:15:50.686 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.686 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-26:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.26", 2020-11-09T18:15:50.686 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:18.528310207Z", 2020-11-09T18:15:50.686 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.687 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.687 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_tUH5PREzBpkbjt0vmvKAgOc", 2020-11-09T18:15:50.687 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.687 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.687 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.687 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_tUH5PREzBpkbjt0vmvKAgOc", 2020-11-09T18:15:50.687 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:50.688 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.688 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.688 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.688 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.688 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.688 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.688 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.689 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945598.543001_154.1", 2020-11-09T18:15:50.689 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.689 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-26:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.26", 2020-11-09T18:15:50.689 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:18.543001147Z", 2020-11-09T18:15:50.689 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.689 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.689 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_tUH5PREzBpkbjt0vmvKAgOc", 2020-11-09T18:15:50.689 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:50.690 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.690 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.690 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_tUH5PREzBpkbjt0vmvKAgOc", 2020-11-09T18:15:50.690 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 3 2020-11-09T18:15:50.690 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.690 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.690 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.691 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.691 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.691 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.691 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.691 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945598.548940_155.1", 2020-11-09T18:15:50.691 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.691 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-26:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.26", 2020-11-09T18:15:50.691 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:18.548940287Z", 2020-11-09T18:15:50.692 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.692 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.692 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_tUH5PREzBpkbjt0vmvKAgOc", 2020-11-09T18:15:50.692 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:50.692 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.692 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.692 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_tUH5PREzBpkbjt0vmvKAgOc", 2020-11-09T18:15:50.693 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 3 2020-11-09T18:15:50.693 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.693 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.693 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.693 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.693 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.694 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.694 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.694 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945600.502084_156.1", 2020-11-09T18:15:50.694 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.694 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-27:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.27", 2020-11-09T18:15:50.694 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:20.502084837Z", 2020-11-09T18:15:50.695 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.695 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.695 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.695 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.695 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.695 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.695 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_ffjKv4uq3V4IpykJfSlfwhB", 2020-11-09T18:15:50.696 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.696 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.696 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.696 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.696 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.696 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.696 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.697 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.697 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945600.507516_158.1", 2020-11-09T18:15:50.697 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.697 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-27:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.27", 2020-11-09T18:15:50.697 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:20.507516916Z", 2020-11-09T18:15:50.697 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.698 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.698 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.698 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.698 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.698 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.698 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_ffjKv4uq3V4IpykJfSlfwhB", 2020-11-09T18:15:50.699 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.699 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.699 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.699 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.699 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.699 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.699 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.700 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.700 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945600.509661_159.1", 2020-11-09T18:15:50.700 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.700 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-27", 2020-11-09T18:15:50.700 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:20.509661301Z", 2020-11-09T18:15:50.700 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.701 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.701 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.701 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.701 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.701 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.701 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_UJQ25D20xb-Xxms2LevItvh", 2020-11-09T18:15:50.701 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.702 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.702 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.702 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.702 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.702 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.702 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.702 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.703 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945600.513944_160.1", 2020-11-09T18:15:50.703 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.703 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-27", 2020-11-09T18:15:50.703 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:20.513944844Z", 2020-11-09T18:15:50.703 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.703 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.703 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.704 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.704 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.704 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.704 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_UJQ25D20xb-Xxms2LevItvh", 2020-11-09T18:15:50.704 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.704 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.704 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.705 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.705 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.705 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.705 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.705 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.705 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945601.434311_161.1", 2020-11-09T18:15:50.705 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.706 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-28:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.28", 2020-11-09T18:15:50.706 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:21.434311607Z", 2020-11-09T18:15:50.706 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.706 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.706 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.706 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.706 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.707 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.707 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_jFhIAKXmQdsa5R34lXgk3pv", 2020-11-09T18:15:50.707 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.707 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.707 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.707 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.707 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.708 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.708 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.708 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.708 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945601.438688_162.1", 2020-11-09T18:15:50.708 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.708 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-28:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.28", 2020-11-09T18:15:50.708 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:21.438688009Z", 2020-11-09T18:15:50.709 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.709 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.709 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.709 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.709 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.709 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.709 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_jFhIAKXmQdsa5R34lXgk3pv", 2020-11-09T18:15:50.710 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.710 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.710 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.710 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.710 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.710 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.710 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.711 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.711 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945601.440791_163.1", 2020-11-09T18:15:50.711 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.711 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-28", 2020-11-09T18:15:50.711 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:21.440791127Z", 2020-11-09T18:15:50.711 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.712 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.712 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.712 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.712 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.712 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.712 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_4xBDSjdegdCsfVupUBmzzRX", 2020-11-09T18:15:50.712 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.713 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.713 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.713 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.713 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.713 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.713 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.714 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.714 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945601.445125_164.1", 2020-11-09T18:15:50.714 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.714 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-28", 2020-11-09T18:15:50.714 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:21.445125336Z", 2020-11-09T18:15:50.714 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.714 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.715 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.715 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.715 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.715 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.715 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_4xBDSjdegdCsfVupUBmzzRX", 2020-11-09T18:15:50.715 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.716 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.716 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.716 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.716 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.716 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.716 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.717 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.717 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945603.023496_165.1", 2020-11-09T18:15:50.717 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.717 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-29:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.29", 2020-11-09T18:15:50.717 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:23.023496753Z", 2020-11-09T18:15:50.717 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.718 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.718 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.718 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.718 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.718 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.719 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_P83ahmV5nZ3tNcs0_lSHbDU", 2020-11-09T18:15:50.719 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.719 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.719 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.719 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.719 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.720 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.720 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.720 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.720 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945603.027781_166.1", 2020-11-09T18:15:50.720 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.720 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-29:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.29", 2020-11-09T18:15:50.720 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:23.027781495Z", 2020-11-09T18:15:50.721 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.721 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.721 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.721 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.721 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.721 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.721 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_P83ahmV5nZ3tNcs0_lSHbDU", 2020-11-09T18:15:50.721 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.722 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.722 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.722 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.722 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.722 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.722 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.722 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.723 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945603.030309_167.1", 2020-11-09T18:15:50.723 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.723 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-29", 2020-11-09T18:15:50.723 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:23.030309393Z", 2020-11-09T18:15:50.723 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.723 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.723 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.724 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.724 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.724 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.724 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_Hhh1A6pFRNzrIunHlFDZwos", 2020-11-09T18:15:50.724 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.724 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.724 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.725 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.725 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.725 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.725 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.725 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.725 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945603.034891_168.1", 2020-11-09T18:15:50.725 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.726 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-29", 2020-11-09T18:15:50.726 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:23.034891461Z", 2020-11-09T18:15:50.726 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.726 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.726 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.726 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.727 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.727 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.727 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_Hhh1A6pFRNzrIunHlFDZwos", 2020-11-09T18:15:50.727 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.727 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.727 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.727 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.728 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.728 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.728 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.728 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.728 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945603.213259_169.1", 2020-11-09T18:15:50.728 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.729 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-27:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.27", 2020-11-09T18:15:50.729 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:23.213259709Z", 2020-11-09T18:15:50.729 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.729 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.729 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_ffjKv4uq3V4IpykJfSlfwhB", 2020-11-09T18:15:50.729 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.730 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.730 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.730 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_ffjKv4uq3V4IpykJfSlfwhB", 2020-11-09T18:15:50.730 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:50.730 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.730 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.731 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.731 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.731 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.731 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.731 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.731 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945603.223295_170.1", 2020-11-09T18:15:50.732 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.732 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-27:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.27", 2020-11-09T18:15:50.732 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:23.223295591Z", 2020-11-09T18:15:50.732 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.732 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.732 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_ffjKv4uq3V4IpykJfSlfwhB", 2020-11-09T18:15:50.732 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.733 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.733 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.733 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_ffjKv4uq3V4IpykJfSlfwhB", 2020-11-09T18:15:50.733 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:50.733 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.733 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.733 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.734 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.734 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.734 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.734 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.734 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945604.018547_171.1", 2020-11-09T18:15:50.734 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.734 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-28:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.28", 2020-11-09T18:15:50.734 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:24.018547219Z", 2020-11-09T18:15:50.735 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.735 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.735 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_jFhIAKXmQdsa5R34lXgk3pv", 2020-11-09T18:15:50.735 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.735 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.735 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.735 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_jFhIAKXmQdsa5R34lXgk3pv", 2020-11-09T18:15:50.736 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:50.736 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.736 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.736 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.736 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.736 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.736 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.737 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.737 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945604.029428_172.1", 2020-11-09T18:15:50.737 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.737 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-28:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.28", 2020-11-09T18:15:50.737 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:24.029428921Z", 2020-11-09T18:15:50.737 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.737 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.738 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_jFhIAKXmQdsa5R34lXgk3pv", 2020-11-09T18:15:50.738 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.738 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.738 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.738 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_jFhIAKXmQdsa5R34lXgk3pv", 2020-11-09T18:15:50.738 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:50.739 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.739 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.739 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.739 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.739 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.739 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.739 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.740 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945604.720659_173.1", 2020-11-09T18:15:50.740 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.740 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-29:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.29", 2020-11-09T18:15:50.740 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:24.720659097Z", 2020-11-09T18:15:50.740 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.740 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.741 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_P83ahmV5nZ3tNcs0_lSHbDU", 2020-11-09T18:15:50.741 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.741 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.741 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.741 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_P83ahmV5nZ3tNcs0_lSHbDU", 2020-11-09T18:15:50.741 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:50.741 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.742 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.742 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.742 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.742 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.742 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.742 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.743 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945604.732717_174.1", 2020-11-09T18:15:50.743 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.743 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-29:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.29", 2020-11-09T18:15:50.743 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:24.732717989Z", 2020-11-09T18:15:50.743 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.743 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.744 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_P83ahmV5nZ3tNcs0_lSHbDU", 2020-11-09T18:15:50.744 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.744 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.744 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.744 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_P83ahmV5nZ3tNcs0_lSHbDU", 2020-11-09T18:15:50.744 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:50.744 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.745 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.745 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.745 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.745 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.745 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.745 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.746 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945605.406938_175.1", 2020-11-09T18:15:50.746 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.746 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-30:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.30", 2020-11-09T18:15:50.746 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:25.406938541Z", 2020-11-09T18:15:50.746 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.746 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.746 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.747 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.747 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.747 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.747 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_2k6PaixusxbzBO-jYw6D0Md", 2020-11-09T18:15:50.747 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.747 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.747 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.748 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.748 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.748 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.748 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.748 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.748 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945605.412027_176.1", 2020-11-09T18:15:50.748 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.749 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-30:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.30", 2020-11-09T18:15:50.749 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:25.412027440Z", 2020-11-09T18:15:50.749 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.749 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.749 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.749 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.749 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.750 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.750 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_2k6PaixusxbzBO-jYw6D0Md", 2020-11-09T18:15:50.750 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.750 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.750 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.750 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.750 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.751 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.751 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.751 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.751 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945605.413981_177.1", 2020-11-09T18:15:50.751 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.751 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-30", 2020-11-09T18:15:50.751 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:25.413981551Z", 2020-11-09T18:15:50.751 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.752 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.752 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.752 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.752 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.752 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.752 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_OVN11Cxzp_HRsHiBhSltQ2E", 2020-11-09T18:15:50.752 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.753 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.753 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.753 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.753 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.753 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.753 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.753 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.754 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945605.419171_178.1", 2020-11-09T18:15:50.754 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.754 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-30", 2020-11-09T18:15:50.754 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:25.419171989Z", 2020-11-09T18:15:50.754 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.754 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.754 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.755 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.755 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.755 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.755 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_OVN11Cxzp_HRsHiBhSltQ2E", 2020-11-09T18:15:50.755 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.755 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.755 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.756 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.756 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.756 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.756 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.756 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.756 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945605.813895_179.1", 2020-11-09T18:15:50.756 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.757 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-31:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.31", 2020-11-09T18:15:50.757 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:25.813895626Z", 2020-11-09T18:15:50.757 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.757 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.757 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.757 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.757 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.757 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.758 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dboGSnUFvBxtlhGOt6TYcuV", 2020-11-09T18:15:50.758 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.758 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.758 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.758 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.758 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.758 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.759 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.759 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.759 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945605.819203_180.1", 2020-11-09T18:15:50.759 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.759 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-31:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.31", 2020-11-09T18:15:50.759 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:25.819203440Z", 2020-11-09T18:15:50.760 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.760 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.760 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.760 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.760 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.760 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.760 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dboGSnUFvBxtlhGOt6TYcuV", 2020-11-09T18:15:50.761 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.761 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.761 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.761 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.761 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.761 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.762 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.762 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.762 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945605.821244_181.1", 2020-11-09T18:15:50.762 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.762 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-31", 2020-11-09T18:15:50.762 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:25.821244081Z", 2020-11-09T18:15:50.763 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.763 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.763 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.763 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.763 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.763 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.763 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_D_zBqUuf_PUMl6KPscj9tcq", 2020-11-09T18:15:50.764 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.764 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.764 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.764 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.764 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.764 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.765 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.765 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.765 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945605.825854_182.1", 2020-11-09T18:15:50.765 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.765 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-31", 2020-11-09T18:15:50.765 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:25.825854764Z", 2020-11-09T18:15:50.765 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.765 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.766 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.766 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.766 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.766 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.766 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_D_zBqUuf_PUMl6KPscj9tcq", 2020-11-09T18:15:50.766 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.766 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.767 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.767 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.767 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.767 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.767 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.767 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.767 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.314889_183.1", 2020-11-09T18:15:50.768 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.768 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-32:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.32", 2020-11-09T18:15:50.768 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.314889240Z", 2020-11-09T18:15:50.768 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.768 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.768 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.769 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.769 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.769 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.769 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_h7iYndUz6ZZeJDRnCAJhq6b", 2020-11-09T18:15:50.769 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.769 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.769 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.769 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.770 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.770 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.770 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.770 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.770 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.319791_184.1", 2020-11-09T18:15:50.770 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.771 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-32:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.32", 2020-11-09T18:15:50.771 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.319791421Z", 2020-11-09T18:15:50.771 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.771 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.771 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.771 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.771 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.771 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.772 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_h7iYndUz6ZZeJDRnCAJhq6b", 2020-11-09T18:15:50.772 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.772 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.772 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.772 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.772 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.773 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.773 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.773 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.773 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.321799_185.1", 2020-11-09T18:15:50.773 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.773 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-32", 2020-11-09T18:15:50.773 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.321799710Z", 2020-11-09T18:15:50.774 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.774 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.774 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.774 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.774 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.774 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.774 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_kfgLDq_uLGT_Ii4AzNlTUJd", 2020-11-09T18:15:50.775 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.775 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.775 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.775 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.775 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.775 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.775 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.776 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.776 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.326129_186.1", 2020-11-09T18:15:50.776 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.776 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-32", 2020-11-09T18:15:50.776 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.326129812Z", 2020-11-09T18:15:50.776 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.776 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.777 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.777 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.777 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.777 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.777 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_kfgLDq_uLGT_Ii4AzNlTUJd", 2020-11-09T18:15:50.777 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.777 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.777 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.778 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.778 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.778 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.778 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.778 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.778 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.351846_187.1", 2020-11-09T18:15:50.778 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.779 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-30:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.30", 2020-11-09T18:15:50.779 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.351846735Z", 2020-11-09T18:15:50.779 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.779 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.779 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_2k6PaixusxbzBO-jYw6D0Md", 2020-11-09T18:15:50.779 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.779 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.780 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.780 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_2k6PaixusxbzBO-jYw6D0Md", 2020-11-09T18:15:50.780 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:50.780 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.780 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.780 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.780 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.780 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.781 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.781 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.781 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.358962_188.1", 2020-11-09T18:15:50.781 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.781 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-30:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.30", 2020-11-09T18:15:50.781 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.358962403Z", 2020-11-09T18:15:50.781 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.782 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.782 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_2k6PaixusxbzBO-jYw6D0Md", 2020-11-09T18:15:50.782 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.782 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.782 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.782 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_2k6PaixusxbzBO-jYw6D0Md", 2020-11-09T18:15:50.782 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:50.783 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.783 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.783 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.783 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.783 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.783 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.783 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.783 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.413199_189.1", 2020-11-09T18:15:50.784 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.784 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-31:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.31", 2020-11-09T18:15:50.784 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.413199050Z", 2020-11-09T18:15:50.784 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.784 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.784 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dboGSnUFvBxtlhGOt6TYcuV", 2020-11-09T18:15:50.784 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.785 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.785 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.785 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dboGSnUFvBxtlhGOt6TYcuV", 2020-11-09T18:15:50.785 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:50.785 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.785 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.785 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.786 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.786 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.786 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.786 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.786 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.423298_190.1", 2020-11-09T18:15:50.786 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.787 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-31:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.31", 2020-11-09T18:15:50.787 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.423298203Z", 2020-11-09T18:15:50.787 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.787 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.787 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dboGSnUFvBxtlhGOt6TYcuV", 2020-11-09T18:15:50.787 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.787 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.788 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.788 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dboGSnUFvBxtlhGOt6TYcuV", 2020-11-09T18:15:50.788 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:50.788 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.788 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.788 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.789 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.789 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.789 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.789 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.789 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.486367_191.1", 2020-11-09T18:15:50.789 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.789 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-32:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.32", 2020-11-09T18:15:50.790 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.486367158Z", 2020-11-09T18:15:50.790 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.790 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.790 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_h7iYndUz6ZZeJDRnCAJhq6b", 2020-11-09T18:15:50.790 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.790 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.791 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.791 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_h7iYndUz6ZZeJDRnCAJhq6b", 2020-11-09T18:15:50.791 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:50.791 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.791 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.791 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.796 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.796 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.796 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.797 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.797 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.497483_192.1", 2020-11-09T18:15:50.797 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.797 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-32:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.32", 2020-11-09T18:15:50.797 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.497483092Z", 2020-11-09T18:15:50.797 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.798 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.798 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_h7iYndUz6ZZeJDRnCAJhq6b", 2020-11-09T18:15:50.798 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.798 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.798 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.798 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_h7iYndUz6ZZeJDRnCAJhq6b", 2020-11-09T18:15:50.798 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:50.799 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.799 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.799 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.799 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.799 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.799 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.799 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.800 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.567936_193.1", 2020-11-09T18:15:50.800 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.800 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-33:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.33", 2020-11-09T18:15:50.800 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.567936942Z", 2020-11-09T18:15:50.800 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.800 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.800 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.801 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.801 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.801 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.801 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dV7Zwb-EMDhOHgweLJlLD79", 2020-11-09T18:15:50.801 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.801 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.801 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.802 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.802 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.802 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.802 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.802 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.802 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.573179_194.1", 2020-11-09T18:15:50.802 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.803 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-33:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.33", 2020-11-09T18:15:50.803 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.573179777Z", 2020-11-09T18:15:50.803 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.803 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.803 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.803 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.803 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.804 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.804 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dV7Zwb-EMDhOHgweLJlLD79", 2020-11-09T18:15:50.804 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.804 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.804 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.804 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.804 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.804 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.805 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.805 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.805 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.575137_195.1", 2020-11-09T18:15:50.805 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.805 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-33", 2020-11-09T18:15:50.805 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.575137143Z", 2020-11-09T18:15:50.805 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.806 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.806 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.806 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.806 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.806 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.806 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "__9pytw2itoqGRYgrLwyaE0v", 2020-11-09T18:15:50.806 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.807 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.807 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.807 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.807 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.807 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.807 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.808 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.808 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.578921_196.1", 2020-11-09T18:15:50.808 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.808 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-33", 2020-11-09T18:15:50.808 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.578921064Z", 2020-11-09T18:15:50.808 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.808 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.809 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.809 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.809 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.809 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.809 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "__9pytw2itoqGRYgrLwyaE0v", 2020-11-09T18:15:50.809 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.810 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.810 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.810 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.810 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.810 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.810 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.810 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.811 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.603950_198.1", 2020-11-09T18:15:50.811 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.811 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-34:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.34", 2020-11-09T18:15:50.811 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.603950696Z", 2020-11-09T18:15:50.811 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.811 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.812 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.812 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.812 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.812 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.812 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_KazzTgKF9mHRl9t-mjqUZIF", 2020-11-09T18:15:50.812 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.812 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.813 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.813 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.813 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.813 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.813 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.813 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.813 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.608737_199.1", 2020-11-09T18:15:50.814 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.814 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-34:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.34", 2020-11-09T18:15:50.814 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.608737440Z", 2020-11-09T18:15:50.814 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.814 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.814 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.815 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.815 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.815 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.815 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_KazzTgKF9mHRl9t-mjqUZIF", 2020-11-09T18:15:50.815 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.815 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.816 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.816 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.816 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.816 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.816 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.816 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.817 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.610541_200.1", 2020-11-09T18:15:50.817 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.817 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-34", 2020-11-09T18:15:50.817 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.610541395Z", 2020-11-09T18:15:50.817 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.817 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.817 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.818 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.818 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.818 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.818 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_QkHzIdBeCUUuKSc9kaJ3NT5", 2020-11-09T18:15:50.818 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.818 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.818 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.819 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.819 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.819 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.819 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.819 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.819 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.614352_201.1", 2020-11-09T18:15:50.819 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.820 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-34", 2020-11-09T18:15:50.820 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.614352129Z", 2020-11-09T18:15:50.820 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.820 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.820 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.820 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.820 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.821 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.821 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_QkHzIdBeCUUuKSc9kaJ3NT5", 2020-11-09T18:15:50.821 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.821 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.821 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.821 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.821 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.822 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.822 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.822 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.822 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.658773_202.1", 2020-11-09T18:15:50.822 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.822 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-35:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.35", 2020-11-09T18:15:50.822 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.658773780Z", 2020-11-09T18:15:50.823 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.823 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.823 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.823 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.823 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.823 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.823 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dAiojqzewqe1JboY85aFSyJ", 2020-11-09T18:15:50.823 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.824 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.824 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.824 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.824 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.824 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.824 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.824 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.825 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.663192_203.1", 2020-11-09T18:15:50.825 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.825 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-35:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.35", 2020-11-09T18:15:50.825 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.663192679Z", 2020-11-09T18:15:50.825 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.825 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.825 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.826 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.826 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.826 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.826 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dAiojqzewqe1JboY85aFSyJ", 2020-11-09T18:15:50.826 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.826 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.827 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.827 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.827 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.827 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.827 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.827 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.827 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.665284_204.1", 2020-11-09T18:15:50.828 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.828 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-35", 2020-11-09T18:15:50.828 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.665284756Z", 2020-11-09T18:15:50.828 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.828 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.828 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.829 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.829 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.829 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.829 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_YIG4M8hwsEikvpJIsO0cCiX", 2020-11-09T18:15:50.829 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.829 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.830 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.830 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.830 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.830 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.830 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.830 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.831 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.671390_205.1", 2020-11-09T18:15:50.831 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.831 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-35", 2020-11-09T18:15:50.831 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.671390803Z", 2020-11-09T18:15:50.831 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.831 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.831 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.832 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.832 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.832 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.832 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_YIG4M8hwsEikvpJIsO0cCiX", 2020-11-09T18:15:50.832 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.832 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.832 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.833 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.833 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.833 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.833 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.833 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.833 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.708610_206.1", 2020-11-09T18:15:50.833 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.834 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-33:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.33", 2020-11-09T18:15:50.834 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.708610357Z", 2020-11-09T18:15:50.834 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.834 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.834 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dV7Zwb-EMDhOHgweLJlLD79", 2020-11-09T18:15:50.834 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.834 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.835 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.835 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dV7Zwb-EMDhOHgweLJlLD79", 2020-11-09T18:15:50.835 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:50.835 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.835 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.835 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.835 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.835 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.836 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.836 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.836 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.713328_207.1", 2020-11-09T18:15:50.836 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.836 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-33:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.33", 2020-11-09T18:15:50.836 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.713328296Z", 2020-11-09T18:15:50.837 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.837 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.837 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dV7Zwb-EMDhOHgweLJlLD79", 2020-11-09T18:15:50.837 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.837 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.837 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.837 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dV7Zwb-EMDhOHgweLJlLD79", 2020-11-09T18:15:50.837 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:50.838 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.838 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.838 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.838 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.838 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.838 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.838 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.839 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945607.760169_210.1", 2020-11-09T18:15:50.839 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.839 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-34:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.34", 2020-11-09T18:15:50.839 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:27.760169945Z", 2020-11-09T18:15:50.839 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.839 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.839 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_KazzTgKF9mHRl9t-mjqUZIF", 2020-11-09T18:15:50.840 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.840 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.840 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.840 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_KazzTgKF9mHRl9t-mjqUZIF", 2020-11-09T18:15:50.840 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:50.840 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.840 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.841 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.841 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.841 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.841 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.841 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.841 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945607.771847_211.1", 2020-11-09T18:15:50.841 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.842 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-34:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.34", 2020-11-09T18:15:50.842 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:27.771847568Z", 2020-11-09T18:15:50.842 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.842 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.842 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_KazzTgKF9mHRl9t-mjqUZIF", 2020-11-09T18:15:50.842 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.843 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.843 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.843 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_KazzTgKF9mHRl9t-mjqUZIF", 2020-11-09T18:15:50.843 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:50.843 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.843 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.843 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.844 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.844 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.844 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.844 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.844 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945609.277502_212.1", 2020-11-09T18:15:50.844 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.844 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-35:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.35", 2020-11-09T18:15:50.845 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:29.277502241Z", 2020-11-09T18:15:50.845 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.845 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.845 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dAiojqzewqe1JboY85aFSyJ", 2020-11-09T18:15:50.845 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.845 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.846 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.846 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dAiojqzewqe1JboY85aFSyJ", 2020-11-09T18:15:50.846 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:50.846 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.846 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.846 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.847 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.847 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.847 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.847 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.847 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945609.288552_213.1", 2020-11-09T18:15:50.847 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.847 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-35:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.35", 2020-11-09T18:15:50.848 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:29.288552016Z", 2020-11-09T18:15:50.848 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.848 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.848 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dAiojqzewqe1JboY85aFSyJ", 2020-11-09T18:15:50.848 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.848 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.849 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.849 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dAiojqzewqe1JboY85aFSyJ", 2020-11-09T18:15:50.849 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:50.849 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.849 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.849 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.850 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.850 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.850 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.850 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.850 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945609.660249_214.1", 2020-11-09T18:15:50.850 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.850 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-36:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.36", 2020-11-09T18:15:50.851 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:29.660249505Z", 2020-11-09T18:15:50.851 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.851 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.851 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.851 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.851 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.851 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.852 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_ozJuqjR3GVh8aGnB2Xh4hR5", 2020-11-09T18:15:50.852 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.852 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.852 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.852 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.852 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.852 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.853 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.853 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.853 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945609.665942_215.1", 2020-11-09T18:15:50.853 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.853 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-36:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.36", 2020-11-09T18:15:50.853 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:29.665942367Z", 2020-11-09T18:15:50.853 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.854 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.854 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.854 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.854 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.854 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.854 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_ozJuqjR3GVh8aGnB2Xh4hR5", 2020-11-09T18:15:50.854 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.855 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.855 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.855 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.855 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.855 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.855 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.855 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.855 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945609.669229_216.1", 2020-11-09T18:15:50.856 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.856 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-36", 2020-11-09T18:15:50.856 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:29.669229416Z", 2020-11-09T18:15:50.856 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.856 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.856 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.857 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.857 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.857 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.857 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_DZ1KbRgq_ERertkZGzu4RA_", 2020-11-09T18:15:50.857 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.857 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.858 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.858 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.858 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.858 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.858 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.858 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.858 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945609.675301_217.1", 2020-11-09T18:15:50.859 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.859 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-36", 2020-11-09T18:15:50.859 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:29.675301279Z", 2020-11-09T18:15:50.859 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.859 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.859 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.859 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.860 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.860 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.860 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_DZ1KbRgq_ERertkZGzu4RA_", 2020-11-09T18:15:50.860 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.860 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.860 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.861 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.861 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.861 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.861 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.861 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.861 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945610.079342_218.1", 2020-11-09T18:15:50.862 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.862 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-37:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.37", 2020-11-09T18:15:50.862 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:30.079342479Z", 2020-11-09T18:15:50.862 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.862 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.863 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.863 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.863 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.863 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.863 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_fx1DINtpIECPzOGaSfs3ucL", 2020-11-09T18:15:50.863 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.863 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.864 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.864 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.864 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.864 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.864 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.864 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.865 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945610.083364_219.1", 2020-11-09T18:15:50.865 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.865 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-37:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.37", 2020-11-09T18:15:50.865 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:30.083364883Z", 2020-11-09T18:15:50.865 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.865 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.866 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.866 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.866 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.866 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.866 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_fx1DINtpIECPzOGaSfs3ucL", 2020-11-09T18:15:50.866 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.866 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.867 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.867 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.867 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.867 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.867 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.867 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.868 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945610.085275_220.1", 2020-11-09T18:15:50.868 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.868 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-37", 2020-11-09T18:15:50.868 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:30.085275759Z", 2020-11-09T18:15:50.868 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.868 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.869 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.869 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.869 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.869 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.869 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_J2nfKVSwwp-9vo2WCGxBAhy", 2020-11-09T18:15:50.869 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.870 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.870 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.870 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.870 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.870 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.870 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.870 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.871 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945610.091583_221.1", 2020-11-09T18:15:50.871 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.871 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-37", 2020-11-09T18:15:50.871 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:30.091583370Z", 2020-11-09T18:15:50.871 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.871 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.871 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.872 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.872 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.872 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.872 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_J2nfKVSwwp-9vo2WCGxBAhy", 2020-11-09T18:15:50.872 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.872 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.872 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.873 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.873 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.873 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.873 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.873 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.873 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945610.137385_222.1", 2020-11-09T18:15:50.873 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.874 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-38:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.38", 2020-11-09T18:15:50.874 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:30.137385505Z", 2020-11-09T18:15:50.874 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.874 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.874 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.874 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.875 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.875 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.875 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_MKdJSqznasfgUfD8pSqNyMN", 2020-11-09T18:15:50.875 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.875 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.875 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.876 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.876 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.876 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.876 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.876 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.876 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945610.141970_223.1", 2020-11-09T18:15:50.876 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.877 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-38:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.38", 2020-11-09T18:15:50.877 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:30.141970597Z", 2020-11-09T18:15:50.877 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.877 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.877 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.877 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.878 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.878 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.878 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_MKdJSqznasfgUfD8pSqNyMN", 2020-11-09T18:15:50.878 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.878 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.878 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.878 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.879 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.879 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.879 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.879 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.879 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945610.144161_224.1", 2020-11-09T18:15:50.879 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.879 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-38", 2020-11-09T18:15:50.880 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:30.144161714Z", 2020-11-09T18:15:50.880 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.880 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.880 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.880 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.880 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.881 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.881 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_Lsx6lFhkZQ0sbnB7SgpHP7X", 2020-11-09T18:15:50.881 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.881 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.881 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.881 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.881 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.882 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.882 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.882 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.882 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945610.148613_225.1", 2020-11-09T18:15:50.882 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.882 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-38", 2020-11-09T18:15:50.883 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:30.148613440Z", 2020-11-09T18:15:50.883 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.883 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.883 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.883 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.883 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.883 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.883 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_Lsx6lFhkZQ0sbnB7SgpHP7X", 2020-11-09T18:15:50.884 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.884 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.884 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.884 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.884 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.884 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.884 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.885 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.885 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945611.763559_226.1", 2020-11-09T18:15:50.885 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.885 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-39:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.39", 2020-11-09T18:15:50.885 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:31.763559546Z", 2020-11-09T18:15:50.885 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.885 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.885 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.886 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.886 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.886 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.886 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_DoCr53MUjfmeA1PTWsLAoZg", 2020-11-09T18:15:50.886 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.886 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.886 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.887 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.887 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.887 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.887 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.887 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.887 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945611.768768_228.1", 2020-11-09T18:15:50.887 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.888 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-39:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.39", 2020-11-09T18:15:50.888 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:31.768768717Z", 2020-11-09T18:15:50.888 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.888 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.888 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.888 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.888 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.889 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.889 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_DoCr53MUjfmeA1PTWsLAoZg", 2020-11-09T18:15:50.889 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.889 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.889 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.889 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.890 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.890 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.890 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.890 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.890 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945611.771007_229.1", 2020-11-09T18:15:50.890 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.891 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-39", 2020-11-09T18:15:50.891 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:31.771007696Z", 2020-11-09T18:15:50.891 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.891 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.891 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.891 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.891 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.892 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.892 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_pGgwgGXPCFeV9dl7j-mkG2D", 2020-11-09T18:15:50.892 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.892 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.892 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.892 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.893 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.893 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.893 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.893 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.893 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945611.776767_230.1", 2020-11-09T18:15:50.893 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.893 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-39", 2020-11-09T18:15:50.894 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:31.776767187Z", 2020-11-09T18:15:50.894 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.894 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.894 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.894 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.894 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.895 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.895 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_pGgwgGXPCFeV9dl7j-mkG2D", 2020-11-09T18:15:50.895 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.895 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.895 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.895 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.895 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.895 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.896 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.896 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.896 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945612.536576_231.1", 2020-11-09T18:15:50.896 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.896 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-40:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.40", 2020-11-09T18:15:50.896 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:32.536576361Z", 2020-11-09T18:15:50.896 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.897 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.897 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.897 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.897 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.897 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.897 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_eRiB6BdOLC_U7ajfFCuPyzq", 2020-11-09T18:15:50.897 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.898 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.898 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.898 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.898 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.898 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.898 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.898 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.898 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945612.542749_232.1", 2020-11-09T18:15:50.899 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.899 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-40:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.40", 2020-11-09T18:15:50.899 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:32.542749842Z", 2020-11-09T18:15:50.899 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.899 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.899 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.899 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.900 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.900 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.900 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_eRiB6BdOLC_U7ajfFCuPyzq", 2020-11-09T18:15:50.900 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.900 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.900 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.900 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.901 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.901 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.901 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.901 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.901 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945612.545398_233.1", 2020-11-09T18:15:50.901 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.902 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-40", 2020-11-09T18:15:50.902 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:32.545398453Z", 2020-11-09T18:15:50.902 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.902 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.902 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.903 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.903 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.903 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.903 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_v-PHMVkQGnPvIMVgQXzEv-e", 2020-11-09T18:15:50.903 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.903 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.904 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.904 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.904 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.904 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.904 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.904 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.905 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945612.550287_234.1", 2020-11-09T18:15:50.905 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.905 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-40", 2020-11-09T18:15:50.905 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:32.550287369Z", 2020-11-09T18:15:50.905 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.905 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.906 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.906 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.906 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.906 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.906 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_v-PHMVkQGnPvIMVgQXzEv-e", 2020-11-09T18:15:50.906 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.907 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.907 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.907 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.907 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.907 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.907 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.907 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.908 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945612.641557_236.1", 2020-11-09T18:15:50.908 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.909 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-41:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.41", 2020-11-09T18:15:50.909 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:32.641557244Z", 2020-11-09T18:15:50.910 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.910 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.910 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.910 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.910 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.910 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.910 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_VbEzzl6PqzRKUgwPXZGxMNK", 2020-11-09T18:15:50.911 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.911 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.911 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.911 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.911 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.911 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.911 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.912 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.912 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945612.646933_237.1", 2020-11-09T18:15:50.912 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.912 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-41:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.41", 2020-11-09T18:15:50.912 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:32.646933508Z", 2020-11-09T18:15:50.912 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.912 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.913 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.913 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.913 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.913 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.913 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_VbEzzl6PqzRKUgwPXZGxMNK", 2020-11-09T18:15:50.913 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.913 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.914 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.914 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.914 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.914 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.914 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.914 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.914 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945612.649001_238.1", 2020-11-09T18:15:50.915 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.915 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-41", 2020-11-09T18:15:50.915 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:32.649001607Z", 2020-11-09T18:15:50.915 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.915 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.915 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.915 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.916 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.916 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.916 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_-QlJbrN7eIOTl-coxNoWB0_", 2020-11-09T18:15:50.916 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.916 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.916 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.916 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.917 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.917 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.917 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.917 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.917 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945612.653005_239.1", 2020-11-09T18:15:50.917 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.918 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-41", 2020-11-09T18:15:50.918 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:32.653005302Z", 2020-11-09T18:15:50.918 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.918 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.918 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.918 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.919 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.919 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.919 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_-QlJbrN7eIOTl-coxNoWB0_", 2020-11-09T18:15:50.919 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.919 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.919 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.920 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.920 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.920 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.920 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.920 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.920 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945563.557787_47.1", 2020-11-09T18:15:50.920 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.921 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-1:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.1", 2020-11-09T18:15:50.921 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:43.557787352Z", 2020-11-09T18:15:50.921 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.921 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.921 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.921 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.922 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.922 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.922 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_5-8ThY0rD5nJEgi4q-i6qW8", 2020-11-09T18:15:50.922 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.922 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.922 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.923 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.923 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.923 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.923 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.923 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.923 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945563.567624_51.1", 2020-11-09T18:15:50.924 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.924 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-1:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.1", 2020-11-09T18:15:50.924 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:43.567624280Z", 2020-11-09T18:15:50.924 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.924 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.924 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.925 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.925 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.925 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.925 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_5-8ThY0rD5nJEgi4q-i6qW8", 2020-11-09T18:15:50.925 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.925 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.925 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.925 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.926 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.926 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.926 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.926 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.926 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945563.569310_52.1", 2020-11-09T18:15:50.926 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.926 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-1", 2020-11-09T18:15:50.927 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:43.569310797Z", 2020-11-09T18:15:50.927 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.927 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.927 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.927 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.927 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.927 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.928 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_KrRBCosiXlBz0ajusgkrpWS", 2020-11-09T18:15:50.928 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.928 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.928 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.928 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.928 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.928 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.929 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.929 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.929 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945563.577890_53.1", 2020-11-09T18:15:50.929 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.929 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-1", 2020-11-09T18:15:50.929 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:43.577890725Z", 2020-11-09T18:15:50.929 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.929 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.930 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.930 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.930 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.930 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.930 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_KrRBCosiXlBz0ajusgkrpWS", 2020-11-09T18:15:50.930 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.930 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.931 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.931 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.931 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.931 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.931 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.931 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.931 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945563.608973_54.1", 2020-11-09T18:15:50.932 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.932 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-2:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.2", 2020-11-09T18:15:50.932 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:43.608973238Z", 2020-11-09T18:15:50.932 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.932 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.932 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.932 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.932 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.933 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.933 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_eZ3F6MzV7ES6qOIMaFzRrOO", 2020-11-09T18:15:50.933 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.933 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.933 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.933 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.933 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.934 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.934 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.934 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.934 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945563.618816_55.1", 2020-11-09T18:15:50.934 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.934 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-2:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.2", 2020-11-09T18:15:50.935 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:43.618816140Z", 2020-11-09T18:15:50.935 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.935 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.935 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.935 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.935 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.935 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.936 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_eZ3F6MzV7ES6qOIMaFzRrOO", 2020-11-09T18:15:50.936 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.936 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.936 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.936 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.936 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.937 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.937 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.937 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.937 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945563.621200_56.1", 2020-11-09T18:15:50.937 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.937 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-2", 2020-11-09T18:15:50.937 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:43.621200541Z", 2020-11-09T18:15:50.938 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.938 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.938 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.938 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.938 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.938 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.939 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_rj9KSIrvtNvXYWU2UYD1Z_m", 2020-11-09T18:15:50.939 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.939 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.939 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.939 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.939 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.940 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.940 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.940 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.940 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945563.627302_57.1", 2020-11-09T18:15:50.940 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.940 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-2", 2020-11-09T18:15:50.940 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:43.627302782Z", 2020-11-09T18:15:50.941 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.941 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.941 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.941 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.941 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.941 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.942 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_rj9KSIrvtNvXYWU2UYD1Z_m", 2020-11-09T18:15:50.942 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.942 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.942 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.942 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.942 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.943 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.943 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.943 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.943 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945567.627267_58.1", 2020-11-09T18:15:50.943 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.943 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-3:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.3", 2020-11-09T18:15:50.943 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:47.627267860Z", 2020-11-09T18:15:50.944 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.944 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.944 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.944 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.944 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.944 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.944 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_TnwqP2_mu3KZnFJOXhnO-DD", 2020-11-09T18:15:50.945 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.945 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.945 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.945 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.945 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.945 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.946 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.946 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.946 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945567.630834_59.1", 2020-11-09T18:15:50.946 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.946 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-3:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.3", 2020-11-09T18:15:50.946 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:47.630834733Z", 2020-11-09T18:15:50.947 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.947 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.947 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.947 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.947 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.947 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.947 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_TnwqP2_mu3KZnFJOXhnO-DD", 2020-11-09T18:15:50.948 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.948 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.948 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.948 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.948 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.948 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.948 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.948 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.949 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945567.632589_60.1", 2020-11-09T18:15:50.949 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.949 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-3", 2020-11-09T18:15:50.949 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:47.632589277Z", 2020-11-09T18:15:50.949 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.949 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.949 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.950 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.950 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.950 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.950 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_MHzcuL8U9yY5bsA8iZvWhTc", 2020-11-09T18:15:50.950 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.950 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.950 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.951 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.951 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.951 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.951 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.951 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.951 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945567.636634_61.1", 2020-11-09T18:15:50.951 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.951 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-3", 2020-11-09T18:15:50.952 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:47.636634661Z", 2020-11-09T18:15:50.952 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.952 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.952 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.952 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.952 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.952 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.953 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_MHzcuL8U9yY5bsA8iZvWhTc", 2020-11-09T18:15:50.953 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.953 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.953 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.953 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.953 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.953 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.954 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.954 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.954 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945570.370676_62.1", 2020-11-09T18:15:50.954 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.954 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-4:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.4", 2020-11-09T18:15:50.954 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:50.370676662Z", 2020-11-09T18:15:50.954 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.955 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.955 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.955 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.955 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.955 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.955 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_ufcC505t7ROkSzsmsL1irho", 2020-11-09T18:15:50.955 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.956 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.956 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.956 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.956 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.956 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.956 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.956 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.957 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945570.375714_64.1", 2020-11-09T18:15:50.957 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.957 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-4:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.4", 2020-11-09T18:15:50.957 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:50.375714519Z", 2020-11-09T18:15:50.957 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.957 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.957 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.958 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.958 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.958 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.958 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_ufcC505t7ROkSzsmsL1irho", 2020-11-09T18:15:50.958 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.958 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.959 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.959 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.959 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.959 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.959 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.959 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.959 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945570.377850_65.1", 2020-11-09T18:15:50.960 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.960 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-4", 2020-11-09T18:15:50.960 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:50.377850670Z", 2020-11-09T18:15:50.960 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.960 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.960 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.961 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.961 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.961 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.961 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_xnR_YIJ8zUxxn61yz_DTUkk", 2020-11-09T18:15:50.961 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.961 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.961 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.962 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.962 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.962 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.962 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.962 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.962 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945570.382459_66.1", 2020-11-09T18:15:50.963 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.963 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-4", 2020-11-09T18:15:50.963 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:50.382459732Z", 2020-11-09T18:15:50.963 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.963 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.963 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.964 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.964 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.964 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.964 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_xnR_YIJ8zUxxn61yz_DTUkk", 2020-11-09T18:15:50.964 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.964 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.964 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.965 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.965 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.965 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.965 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.965 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.965 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945570.406070_67.1", 2020-11-09T18:15:50.966 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.966 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-5:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.5", 2020-11-09T18:15:50.966 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:50.406070925Z", 2020-11-09T18:15:50.966 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.966 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.966 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.967 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.967 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.967 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.967 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_UShZGGBB4kBaxRzXtVmZBUW", 2020-11-09T18:15:50.967 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.967 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.967 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.968 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.968 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.968 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.968 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.968 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.968 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945570.410945_68.1", 2020-11-09T18:15:50.968 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.969 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-5:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.5", 2020-11-09T18:15:50.969 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:50.410945255Z", 2020-11-09T18:15:50.969 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.969 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.969 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.969 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.969 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.970 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.970 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_UShZGGBB4kBaxRzXtVmZBUW", 2020-11-09T18:15:50.970 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.970 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.970 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.970 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.970 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.971 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.971 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.971 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.971 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945570.412796_69.1", 2020-11-09T18:15:50.971 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.971 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-5", 2020-11-09T18:15:50.971 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:50.412796913Z", 2020-11-09T18:15:50.972 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.972 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.972 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.972 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.972 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.972 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.972 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_5bx-BH8ijLijK0Hb5tRXntZ", 2020-11-09T18:15:50.972 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.973 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.973 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.973 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.973 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.973 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.973 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.973 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.974 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945570.416832_70.1", 2020-11-09T18:15:50.974 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.974 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-5", 2020-11-09T18:15:50.974 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:50.416832729Z", 2020-11-09T18:15:50.974 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.974 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.974 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.975 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.975 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.975 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.975 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_5bx-BH8ijLijK0Hb5tRXntZ", 2020-11-09T18:15:50.975 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.975 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.975 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.975 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.976 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.976 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.976 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.976 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.976 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945570.461105_71.1", 2020-11-09T18:15:50.976 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.976 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-6:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.6", 2020-11-09T18:15:50.977 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:50.461105755Z", 2020-11-09T18:15:50.977 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.977 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.977 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.977 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.977 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.977 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.978 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_qWlHsdsWYJHc0mQkDC3vzYx", 2020-11-09T18:15:50.978 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.978 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.978 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.978 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.978 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.978 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.979 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.979 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.979 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945570.473537_72.1", 2020-11-09T18:15:50.979 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.979 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-6:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.6", 2020-11-09T18:15:50.979 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:50.473537938Z", 2020-11-09T18:15:50.980 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.980 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.980 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.980 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.980 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.980 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.980 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_qWlHsdsWYJHc0mQkDC3vzYx", 2020-11-09T18:15:50.981 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.981 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.981 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.981 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.981 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.981 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.982 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.982 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.982 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945570.477860_73.1", 2020-11-09T18:15:50.982 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.982 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-6", 2020-11-09T18:15:50.982 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:50.477860682Z", 2020-11-09T18:15:50.983 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.983 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.983 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.983 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.983 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.983 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.983 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dmuyyeGa1O--zFbpSa0SA8D", 2020-11-09T18:15:50.984 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.984 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.984 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.984 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.984 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.984 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.985 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.985 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.985 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945570.489122_74.1", 2020-11-09T18:15:50.985 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.985 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-6", 2020-11-09T18:15:50.985 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:50.489122994Z", 2020-11-09T18:15:50.986 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.986 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.986 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.986 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.986 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.986 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.987 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dmuyyeGa1O--zFbpSa0SA8D", 2020-11-09T18:15:50.987 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.987 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.987 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.987 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.987 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.987 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.988 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.988 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.988 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945579.353942_75.1", 2020-11-09T18:15:50.988 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.988 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-7:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.7", 2020-11-09T18:15:50.988 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:59.353942489Z", 2020-11-09T18:15:50.989 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.989 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.989 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.989 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.989 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.989 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.990 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_GIkCDw3GWmDRO3-A0auRtxB", 2020-11-09T18:15:50.990 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.990 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.990 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.990 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.990 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.990 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.991 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.991 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.991 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945579.359041_78.1", 2020-11-09T18:15:50.991 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.991 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-7:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.7", 2020-11-09T18:15:50.991 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:59.359041450Z", 2020-11-09T18:15:50.991 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.992 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.992 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.992 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.992 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.992 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.992 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_GIkCDw3GWmDRO3-A0auRtxB", 2020-11-09T18:15:50.992 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.992 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.993 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.993 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.993 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.993 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.993 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.993 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.993 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945579.361526_79.1", 2020-11-09T18:15:50.994 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.994 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-7", 2020-11-09T18:15:50.994 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:59.361526900Z", 2020-11-09T18:15:50.994 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.994 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.994 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.994 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.995 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.995 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.995 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_euurhSSx6WmV693QUVvezHc", 2020-11-09T18:15:50.995 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.995 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.995 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.995 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:50.996 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.996 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.996 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.996 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.996 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945579.367322_80.1", 2020-11-09T18:15:50.996 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:50.996 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-7", 2020-11-09T18:15:50.996 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:59.367322143Z", 2020-11-09T18:15:50.997 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:50.997 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:50.997 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:50.997 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:50.997 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.997 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:50.998 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_euurhSSx6WmV693QUVvezHc", 2020-11-09T18:15:50.998 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:50.998 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.998 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:50.998 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:50.998 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.999 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:50.999 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:50.999 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:50.999 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945579.395388_81.1", 2020-11-09T18:15:50.999 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:50.999 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-8:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.8", 2020-11-09T18:15:51.000 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:59.395388143Z", 2020-11-09T18:15:51.000 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:51.000 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:51.000 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:51.000 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:51.000 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:51.000 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:51.001 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_yBzYlhlGW2rcaJy91e-7eds", 2020-11-09T18:15:51.001 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:51.001 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:51.001 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:51.001 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:51.001 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:51.002 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:51.002 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:51.002 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:51.002 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945579.400026_82.1", 2020-11-09T18:15:51.002 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.002 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-8:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.8", 2020-11-09T18:15:51.002 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:59.400026920Z", 2020-11-09T18:15:51.003 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:51.003 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:51.003 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:51.003 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:51.003 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:51.003 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:51.004 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_yBzYlhlGW2rcaJy91e-7eds", 2020-11-09T18:15:51.004 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:51.004 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:51.004 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:51.004 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:51.004 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:51.004 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:51.005 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:51.005 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:51.005 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945579.402183_83.1", 2020-11-09T18:15:51.005 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:51.005 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-8", 2020-11-09T18:15:51.005 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:59.402183037Z", 2020-11-09T18:15:51.006 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:51.006 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:51.006 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:51.006 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:51.006 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:51.006 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:51.007 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_OGdV9KW-t2xB2NYjc8pNB2U", 2020-11-09T18:15:51.007 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:51.007 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:51.007 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:51.007 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:51.007 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:51.007 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:51.008 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:51.008 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:51.008 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945579.406495_84.1", 2020-11-09T18:15:51.008 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:51.008 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-8", 2020-11-09T18:15:51.008 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:59.406495328Z", 2020-11-09T18:15:51.008 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:51.008 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:51.009 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:51.009 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:51.009 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:51.009 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:51.009 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_OGdV9KW-t2xB2NYjc8pNB2U", 2020-11-09T18:15:51.009 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:51.009 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:51.010 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:51.010 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:51.010 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:51.010 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:51.010 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:51.010 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:51.010 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945579.450093_85.1", 2020-11-09T18:15:51.011 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.011 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-9:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.9", 2020-11-09T18:15:51.011 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:59.450093555Z", 2020-11-09T18:15:51.011 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:51.011 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:51.011 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:51.011 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:51.012 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:51.012 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:51.012 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_sx-mxXXsFhATfbPuJXuh1It", 2020-11-09T18:15:51.012 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:51.012 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:51.012 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:51.012 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:51.013 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:51.013 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:51.013 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:51.013 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:51.013 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945579.455372_86.1", 2020-11-09T18:15:51.013 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.013 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-9:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.9", 2020-11-09T18:15:51.014 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:59.455372433Z", 2020-11-09T18:15:51.014 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:51.014 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:51.014 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:51.014 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:51.014 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:51.014 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:51.015 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_sx-mxXXsFhATfbPuJXuh1It", 2020-11-09T18:15:51.015 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:51.015 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:51.015 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:51.015 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:51.015 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:51.015 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:51.015 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:51.016 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:51.016 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945579.459352_87.1", 2020-11-09T18:15:51.016 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:51.016 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-9", 2020-11-09T18:15:51.016 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:59.459352980Z", 2020-11-09T18:15:51.016 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:51.016 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:51.016 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:51.017 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:51.017 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:51.017 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:51.017 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_CehTzP8PVTogocbVBJZMUE-", 2020-11-09T18:15:51.017 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:51.017 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:51.018 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:51.018 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:51.018 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:51.018 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:51.018 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:51.018 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:51.018 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945579.467305_88.1", 2020-11-09T18:15:51.019 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:51.019 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-9", 2020-11-09T18:15:51.019 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:59.467305227Z", 2020-11-09T18:15:51.019 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:51.019 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:51.019 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:51.020 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:51.020 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:51.020 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:51.020 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_CehTzP8PVTogocbVBJZMUE-", 2020-11-09T18:15:51.020 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:51.021 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:51.021 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:51.021 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:51.021 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:51.021 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:51.021 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:51.021 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:51.022 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945582.240772_89.1", 2020-11-09T18:15:51.022 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:51.022 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-7", 2020-11-09T18:15:51.022 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:02.240772877Z", 2020-11-09T18:15:51.022 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:51.022 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:51.023 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_euurhSSx6WmV693QUVvezHc", 2020-11-09T18:15:51.023 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:51.023 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:51.023 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:51.023 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_euurhSSx6WmV693QUVvezHc", 2020-11-09T18:15:51.023 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:51.024 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:51.024 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:51.024 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "remove" 2020-11-09T18:15:51.024 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:51.024 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:51.024 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:51.024 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:51.025 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945582.251051_91.1", 2020-11-09T18:15:51.025 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:51.025 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-7", 2020-11-09T18:15:51.025 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:02.251051719Z", 2020-11-09T18:15:51.025 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:51.025 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:51.026 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_euurhSSx6WmV693QUVvezHc", 2020-11-09T18:15:51.026 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:51.026 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:51.026 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:51.026 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_euurhSSx6WmV693QUVvezHc", 2020-11-09T18:15:51.026 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:51.026 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:51.027 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:51.027 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:51.027 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:51.027 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:51.027 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:51.027 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:51.027 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945582.313479_92.1", 2020-11-09T18:15:51.028 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:51.028 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-8", 2020-11-09T18:15:51.028 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:02.313479625Z", 2020-11-09T18:15:51.028 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:51.028 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:51.028 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_OGdV9KW-t2xB2NYjc8pNB2U", 2020-11-09T18:15:51.028 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:51.028 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:51.029 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:51.029 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_OGdV9KW-t2xB2NYjc8pNB2U", 2020-11-09T18:15:51.029 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:51.029 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:51.029 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:51.029 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "remove" 2020-11-09T18:15:51.029 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:51.030 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:51.030 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:51.030 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:51.030 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945582.321374_93.1", 2020-11-09T18:15:51.030 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:51.030 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-8", 2020-11-09T18:15:51.030 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:02.321374298Z", 2020-11-09T18:15:51.031 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:51.031 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:51.031 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_OGdV9KW-t2xB2NYjc8pNB2U", 2020-11-09T18:15:51.031 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:51.031 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:51.031 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:51.031 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_OGdV9KW-t2xB2NYjc8pNB2U", 2020-11-09T18:15:51.031 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:51.032 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:51.032 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:51.032 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:51.032 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:51.032 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:51.032 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:51.032 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:51.033 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945582.425686_94.1", 2020-11-09T18:15:51.033 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:51.033 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-9", 2020-11-09T18:15:51.033 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:02.425686180Z", 2020-11-09T18:15:51.033 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:51.033 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:51.033 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_CehTzP8PVTogocbVBJZMUE-", 2020-11-09T18:15:51.034 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:51.034 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:51.034 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:51.034 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_CehTzP8PVTogocbVBJZMUE-", 2020-11-09T18:15:51.034 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:51.034 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:51.034 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:51.035 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "remove" 2020-11-09T18:15:51.035 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:51.035 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:51.035 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:51.035 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:51.035 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945582.432939_95.1", 2020-11-09T18:15:51.036 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:51.036 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-9", 2020-11-09T18:15:51.036 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:02.432939430Z", 2020-11-09T18:15:51.036 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:51.036 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:51.036 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_CehTzP8PVTogocbVBJZMUE-", 2020-11-09T18:15:51.037 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:51.037 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:51.037 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:51.037 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_CehTzP8PVTogocbVBJZMUE-", 2020-11-09T18:15:51.037 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:51.037 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:51.037 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:51.038 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:51.038 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:51.038 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:51.038 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:51.038 INFO:teuthology.orchestra.run.smithi051.stdout:] 2020-11-09T18:15:51.041 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:51.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:51.042 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:15:51.082 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:15:51.082 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:15:51.320 INFO:teuthology.orchestra.run.smithi078.stdout:[ 2020-11-09T18:15:51.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.320 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945439.484659_48.1", 2020-11-09T18:15:51.320 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "user", 2020-11-09T18:15:51.320 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rgw-multisite-test-user", 2020-11-09T18:15:51.321 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:10:39.484659067Z", 2020-11-09T18:15:51.321 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.321 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.321 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.321 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.322 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.322 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_qxUmJfyn2gLxVOOHvE_t0VQ", 2020-11-09T18:15:51.322 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.323 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.323 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.323 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.323 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.323 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945439.492888_50.1", 2020-11-09T18:15:51.324 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "user", 2020-11-09T18:15:51.324 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rgw-multisite-test-user", 2020-11-09T18:15:51.324 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:10:39.492888346Z", 2020-11-09T18:15:51.324 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.324 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.324 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.324 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.325 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.325 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_qxUmJfyn2gLxVOOHvE_t0VQ", 2020-11-09T18:15:51.325 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.325 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.325 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.325 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.326 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.326 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945586.256023_51.1", 2020-11-09T18:15:51.326 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.326 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-10:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.10", 2020-11-09T18:15:51.327 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:06.256023190Z", 2020-11-09T18:15:51.327 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.327 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.327 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.327 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.327 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.327 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.327 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_xk1WELKUHZIqmUd5oTGWrEo", 2020-11-09T18:15:51.328 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.328 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.328 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.328 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.328 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.329 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.329 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945586.261184_59.1", 2020-11-09T18:15:51.329 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.329 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-10:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.10", 2020-11-09T18:15:51.329 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:06.261184780Z", 2020-11-09T18:15:51.329 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.330 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.330 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.330 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.330 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.331 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_xk1WELKUHZIqmUd5oTGWrEo", 2020-11-09T18:15:51.331 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.331 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.331 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.331 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.331 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.332 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.332 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945586.263844_60.1", 2020-11-09T18:15:51.332 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.332 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-10", 2020-11-09T18:15:51.332 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:06.263844660Z", 2020-11-09T18:15:51.332 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.333 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.333 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.333 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.333 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.333 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_p-7-bX-sO2hh9J_2Nonu8rp", 2020-11-09T18:15:51.333 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.334 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.334 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.334 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.335 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.335 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.335 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945586.268581_61.1", 2020-11-09T18:15:51.336 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.336 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-10", 2020-11-09T18:15:51.336 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:06.268581555Z", 2020-11-09T18:15:51.336 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.336 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.337 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.337 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.337 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.337 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_p-7-bX-sO2hh9J_2Nonu8rp", 2020-11-09T18:15:51.337 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.338 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.338 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.338 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.338 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.338 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.338 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945587.202762_62.1", 2020-11-09T18:15:51.339 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.339 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-11:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.11", 2020-11-09T18:15:51.339 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:07.202762446Z", 2020-11-09T18:15:51.339 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.339 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.339 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.339 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.340 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.340 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_5opGpz-RxCA9co32z-sLP0e", 2020-11-09T18:15:51.340 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.340 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.340 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.340 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.341 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.341 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.341 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.341 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945587.207905_63.1", 2020-11-09T18:15:51.341 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.341 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-11:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.11", 2020-11-09T18:15:51.341 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:07.207905728Z", 2020-11-09T18:15:51.342 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.342 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.342 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.342 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.342 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.342 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_5opGpz-RxCA9co32z-sLP0e", 2020-11-09T18:15:51.343 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.343 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.343 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.343 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.343 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.344 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945587.210891_64.1", 2020-11-09T18:15:51.344 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.344 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-11", 2020-11-09T18:15:51.344 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:07.210891570Z", 2020-11-09T18:15:51.344 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.344 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.345 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.345 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.345 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.345 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_QiNkPq8a5F1hoeuAtqCxHNQ", 2020-11-09T18:15:51.345 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.346 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.346 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.346 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.346 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.346 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945587.216871_65.1", 2020-11-09T18:15:51.347 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.347 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-11", 2020-11-09T18:15:51.347 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:07.216871948Z", 2020-11-09T18:15:51.347 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.347 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.347 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.347 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.348 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.348 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_QiNkPq8a5F1hoeuAtqCxHNQ", 2020-11-09T18:15:51.348 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.348 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.348 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.349 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.349 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.349 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945588.126275_66.1", 2020-11-09T18:15:51.349 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.349 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-12:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.12", 2020-11-09T18:15:51.350 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:08.126275141Z", 2020-11-09T18:15:51.350 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.350 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.350 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.350 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.350 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.351 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_7WQjdW6VYC3fP3T1DUBlsCk", 2020-11-09T18:15:51.351 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.351 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.351 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.351 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.352 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.352 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.352 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945588.133515_67.1", 2020-11-09T18:15:51.352 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.353 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-12:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.12", 2020-11-09T18:15:51.353 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:08.133515560Z", 2020-11-09T18:15:51.353 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.353 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.353 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.353 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.354 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.354 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_7WQjdW6VYC3fP3T1DUBlsCk", 2020-11-09T18:15:51.354 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.354 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.354 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.354 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.355 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.355 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945588.135642_68.1", 2020-11-09T18:15:51.355 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.355 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-12", 2020-11-09T18:15:51.355 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:08.135642294Z", 2020-11-09T18:15:51.356 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.356 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.356 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.356 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.356 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.357 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.357 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_PdQ7asc_0k_3_XotzK7eatQ", 2020-11-09T18:15:51.357 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.357 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.357 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.358 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.358 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.358 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.358 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945588.143746_69.1", 2020-11-09T18:15:51.359 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.359 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-12", 2020-11-09T18:15:51.359 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:08.143746572Z", 2020-11-09T18:15:51.359 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.359 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.359 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.360 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.360 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.360 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_PdQ7asc_0k_3_XotzK7eatQ", 2020-11-09T18:15:51.360 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.360 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.361 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.361 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.361 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.361 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.361 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.361 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945588.381357_70.1", 2020-11-09T18:15:51.362 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.362 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-13:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.13", 2020-11-09T18:15:51.362 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:08.381357113Z", 2020-11-09T18:15:51.362 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.362 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.362 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.362 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.363 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.363 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_jRsLDHCkxygm_NRmusaemxh", 2020-11-09T18:15:51.363 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.363 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.363 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.364 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.364 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.364 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.364 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945588.386292_71.1", 2020-11-09T18:15:51.364 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.364 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-13:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.13", 2020-11-09T18:15:51.364 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:08.386292596Z", 2020-11-09T18:15:51.365 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.365 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.365 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.365 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.365 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.366 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_jRsLDHCkxygm_NRmusaemxh", 2020-11-09T18:15:51.366 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.366 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.366 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.366 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.366 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.367 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.367 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945588.391790_72.1", 2020-11-09T18:15:51.367 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.367 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-13", 2020-11-09T18:15:51.368 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:08.391790067Z", 2020-11-09T18:15:51.368 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.368 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.368 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.368 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.368 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.369 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_1zAxS7Eu4GCsgwIw7eCrrnz", 2020-11-09T18:15:51.369 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.369 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.369 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.369 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.369 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.370 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.370 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945588.397490_73.1", 2020-11-09T18:15:51.370 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.370 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-13", 2020-11-09T18:15:51.370 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:08.397490757Z", 2020-11-09T18:15:51.370 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.371 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.371 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.371 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.371 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.371 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_1zAxS7Eu4GCsgwIw7eCrrnz", 2020-11-09T18:15:51.371 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.372 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.372 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.372 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.372 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.372 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945589.380394_74.1", 2020-11-09T18:15:51.373 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.373 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-14:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.14", 2020-11-09T18:15:51.374 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:09.380394647Z", 2020-11-09T18:15:51.374 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.375 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.375 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.375 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.376 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.376 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_ZoR7I5lOnVtZkMZ3aHO24Vp", 2020-11-09T18:15:51.377 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.377 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.378 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.379 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.380 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.380 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945589.385117_75.1", 2020-11-09T18:15:51.380 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.381 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-14:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.14", 2020-11-09T18:15:51.381 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:09.385117237Z", 2020-11-09T18:15:51.381 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.381 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.381 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.381 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.381 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.382 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_ZoR7I5lOnVtZkMZ3aHO24Vp", 2020-11-09T18:15:51.382 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.382 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.382 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.382 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.382 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.383 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945589.387155_76.1", 2020-11-09T18:15:51.383 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.383 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-14", 2020-11-09T18:15:51.383 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:09.387155581Z", 2020-11-09T18:15:51.383 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.384 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.384 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.384 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.384 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.384 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_iQYh1hCLyur_paDmK5MzSiw", 2020-11-09T18:15:51.384 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.385 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.385 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.385 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.385 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.385 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945589.391251_77.1", 2020-11-09T18:15:51.386 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.386 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-14", 2020-11-09T18:15:51.386 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:09.391251540Z", 2020-11-09T18:15:51.386 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.386 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.386 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.387 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.387 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.387 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_iQYh1hCLyur_paDmK5MzSiw", 2020-11-09T18:15:51.387 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.387 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.387 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.388 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.388 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.388 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.388 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945589.523751_78.1", 2020-11-09T18:15:51.388 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.388 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-15:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.15", 2020-11-09T18:15:51.389 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:09.523751440Z", 2020-11-09T18:15:51.389 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.389 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.389 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.389 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.389 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.390 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_KVNlDu4zRBEGnM1a31uWniG", 2020-11-09T18:15:51.390 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.390 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.390 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.390 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.390 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.391 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945589.534160_79.1", 2020-11-09T18:15:51.391 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.391 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-15:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.15", 2020-11-09T18:15:51.391 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:09.534160171Z", 2020-11-09T18:15:51.391 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.392 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.392 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.392 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.392 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.392 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_KVNlDu4zRBEGnM1a31uWniG", 2020-11-09T18:15:51.392 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.393 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.393 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.393 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.393 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.393 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.393 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.393 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.393 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945589.536862_80.1", 2020-11-09T18:15:51.394 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.394 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-15", 2020-11-09T18:15:51.394 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:09.536862933Z", 2020-11-09T18:15:51.394 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.394 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.394 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.394 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.395 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.395 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_Cq6C5mA4ASiylyTIC5SMBlP", 2020-11-09T18:15:51.395 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.395 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.395 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.396 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.396 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.396 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.396 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945589.541920_81.1", 2020-11-09T18:15:51.396 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.396 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-15", 2020-11-09T18:15:51.396 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:09.541920734Z", 2020-11-09T18:15:51.397 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.397 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.397 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.397 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.397 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.397 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_Cq6C5mA4ASiylyTIC5SMBlP", 2020-11-09T18:15:51.398 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.398 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.398 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.398 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.398 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.398 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.398 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.399 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.399 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945590.011767_82.1", 2020-11-09T18:15:51.399 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.399 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-16:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.16", 2020-11-09T18:15:51.399 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:10.011767329Z", 2020-11-09T18:15:51.399 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.399 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.399 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.400 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.400 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.400 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_fJsd-7NZirDCEodHfdpY9oA", 2020-11-09T18:15:51.400 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.400 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.401 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.401 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.401 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.401 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.401 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945590.016722_83.1", 2020-11-09T18:15:51.401 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.401 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-16:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.16", 2020-11-09T18:15:51.402 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:10.016722284Z", 2020-11-09T18:15:51.402 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.402 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.402 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.402 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.402 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.403 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_fJsd-7NZirDCEodHfdpY9oA", 2020-11-09T18:15:51.403 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.403 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.403 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.403 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.403 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.404 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945590.019556_84.1", 2020-11-09T18:15:51.404 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.404 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-16", 2020-11-09T18:15:51.404 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:10.019556366Z", 2020-11-09T18:15:51.404 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.404 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.405 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.405 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.405 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.405 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_83VseKNDEzwg2zXJyAxVSNV", 2020-11-09T18:15:51.405 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.405 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.406 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.406 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.406 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.406 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.406 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945590.028620_85.1", 2020-11-09T18:15:51.406 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.407 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-16", 2020-11-09T18:15:51.407 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:10.028620030Z", 2020-11-09T18:15:51.407 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.407 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.407 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.407 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.407 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.408 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_83VseKNDEzwg2zXJyAxVSNV", 2020-11-09T18:15:51.408 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.408 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.408 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.408 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.408 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.409 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.409 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945590.059189_86.1", 2020-11-09T18:15:51.409 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.409 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-17:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.17", 2020-11-09T18:15:51.409 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:10.059189312Z", 2020-11-09T18:15:51.409 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.410 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.410 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.410 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.410 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.410 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_nhpbrE0BaTecbpBGZZxkC_9", 2020-11-09T18:15:51.410 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.411 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.411 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.411 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.411 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.411 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945590.065317_87.1", 2020-11-09T18:15:51.412 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.412 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-17:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.17", 2020-11-09T18:15:51.412 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:10.065317701Z", 2020-11-09T18:15:51.412 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.412 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.412 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.412 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.413 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.413 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_nhpbrE0BaTecbpBGZZxkC_9", 2020-11-09T18:15:51.413 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.413 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.413 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.413 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.414 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.414 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.414 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.414 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945590.068336_88.1", 2020-11-09T18:15:51.414 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.414 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-17", 2020-11-09T18:15:51.414 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:10.068336838Z", 2020-11-09T18:15:51.415 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.415 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.415 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.415 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.415 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.415 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_DZ4ozYh7iBiXpSqfO2MFhvn", 2020-11-09T18:15:51.416 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.416 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.416 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.416 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.416 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.417 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.417 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945590.076749_89.1", 2020-11-09T18:15:51.417 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.417 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-17", 2020-11-09T18:15:51.417 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:10.076749917Z", 2020-11-09T18:15:51.417 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.417 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.417 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.418 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.418 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.418 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_DZ4ozYh7iBiXpSqfO2MFhvn", 2020-11-09T18:15:51.418 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.418 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.419 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.419 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.419 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.419 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945590.120726_90.1", 2020-11-09T18:15:51.419 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.420 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-18:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.18", 2020-11-09T18:15:51.420 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:10.120726234Z", 2020-11-09T18:15:51.420 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.420 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.420 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.420 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.420 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.421 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_R8b8xLVdGNeTGDdC42EuEfE", 2020-11-09T18:15:51.421 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.421 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.421 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.421 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.421 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.422 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.422 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945590.127213_91.1", 2020-11-09T18:15:51.422 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.422 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-18:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.18", 2020-11-09T18:15:51.422 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:10.127213093Z", 2020-11-09T18:15:51.422 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.423 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.423 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.423 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.423 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.423 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_R8b8xLVdGNeTGDdC42EuEfE", 2020-11-09T18:15:51.423 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.424 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.424 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.424 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.424 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.424 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.424 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945590.129116_92.1", 2020-11-09T18:15:51.425 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.425 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-18", 2020-11-09T18:15:51.425 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:10.129116612Z", 2020-11-09T18:15:51.425 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.425 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.425 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.425 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.426 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.426 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_jvrbBsWxGWFBmAxvo-GL8oI", 2020-11-09T18:15:51.426 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.426 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.426 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.426 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.427 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.427 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945590.133065_93.1", 2020-11-09T18:15:51.427 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.427 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-18", 2020-11-09T18:15:51.427 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:10.133065786Z", 2020-11-09T18:15:51.428 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.428 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.428 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.428 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.428 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.428 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_jvrbBsWxGWFBmAxvo-GL8oI", 2020-11-09T18:15:51.428 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.429 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.429 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.429 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.429 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.429 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.429 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945590.156864_94.1", 2020-11-09T18:15:51.430 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.430 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-16:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.16", 2020-11-09T18:15:51.430 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:10.156864142Z", 2020-11-09T18:15:51.430 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.430 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.430 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_fJsd-7NZirDCEodHfdpY9oA", 2020-11-09T18:15:51.430 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.431 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.431 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_fJsd-7NZirDCEodHfdpY9oA", 2020-11-09T18:15:51.431 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.431 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.431 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.432 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.432 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.432 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945590.163596_95.1", 2020-11-09T18:15:51.432 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.432 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-16:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.16", 2020-11-09T18:15:51.432 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:10.163596031Z", 2020-11-09T18:15:51.433 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.433 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.433 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_fJsd-7NZirDCEodHfdpY9oA", 2020-11-09T18:15:51.433 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.433 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.433 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_fJsd-7NZirDCEodHfdpY9oA", 2020-11-09T18:15:51.434 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.434 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.434 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.434 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.434 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.434 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.435 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945590.168397_96.1", 2020-11-09T18:15:51.435 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.435 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-17:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.17", 2020-11-09T18:15:51.435 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:10.168397876Z", 2020-11-09T18:15:51.435 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.435 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.435 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_nhpbrE0BaTecbpBGZZxkC_9", 2020-11-09T18:15:51.436 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.436 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.436 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_nhpbrE0BaTecbpBGZZxkC_9", 2020-11-09T18:15:51.436 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.436 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.437 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.437 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.437 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.438 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945590.178873_97.1", 2020-11-09T18:15:51.438 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.438 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-17:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.17", 2020-11-09T18:15:51.438 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:10.178873937Z", 2020-11-09T18:15:51.438 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.438 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.439 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_nhpbrE0BaTecbpBGZZxkC_9", 2020-11-09T18:15:51.439 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.439 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.439 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_nhpbrE0BaTecbpBGZZxkC_9", 2020-11-09T18:15:51.440 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.440 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.440 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.440 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.440 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.441 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945590.190493_98.1", 2020-11-09T18:15:51.441 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.441 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-18:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.18", 2020-11-09T18:15:51.441 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:10.190493784Z", 2020-11-09T18:15:51.441 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.441 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.442 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_R8b8xLVdGNeTGDdC42EuEfE", 2020-11-09T18:15:51.442 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.442 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.442 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_R8b8xLVdGNeTGDdC42EuEfE", 2020-11-09T18:15:51.442 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.443 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.443 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.443 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.443 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.443 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945590.199376_99.1", 2020-11-09T18:15:51.443 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.444 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-18:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.18", 2020-11-09T18:15:51.444 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:10.199376961Z", 2020-11-09T18:15:51.444 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.444 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.444 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_R8b8xLVdGNeTGDdC42EuEfE", 2020-11-09T18:15:51.444 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.445 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.445 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_R8b8xLVdGNeTGDdC42EuEfE", 2020-11-09T18:15:51.445 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.445 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.445 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.445 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.446 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.446 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945592.270499_100.1", 2020-11-09T18:15:51.446 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.446 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-19:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.19", 2020-11-09T18:15:51.446 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:12.270499729Z", 2020-11-09T18:15:51.446 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.447 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.447 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.447 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.447 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.447 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_E6pXupKMFQLC75ImCoCkV74", 2020-11-09T18:15:51.447 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.448 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.448 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.448 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.448 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.448 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945592.275423_102.1", 2020-11-09T18:15:51.449 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.449 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-19:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.19", 2020-11-09T18:15:51.449 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:12.275423973Z", 2020-11-09T18:15:51.449 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.449 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.449 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.449 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.450 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.450 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_E6pXupKMFQLC75ImCoCkV74", 2020-11-09T18:15:51.450 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.450 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.450 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.451 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.451 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.451 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945592.278527_103.1", 2020-11-09T18:15:51.451 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.451 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-19", 2020-11-09T18:15:51.451 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:12.278527677Z", 2020-11-09T18:15:51.452 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.452 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.452 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.452 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.452 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.452 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_OGzY_hYHGuuTc6_1SixJvo6", 2020-11-09T18:15:51.453 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.453 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.453 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.453 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.453 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.454 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945592.284263_104.1", 2020-11-09T18:15:51.454 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.454 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-19", 2020-11-09T18:15:51.454 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:12.284263022Z", 2020-11-09T18:15:51.454 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.454 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.455 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.455 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.455 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.455 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_OGzY_hYHGuuTc6_1SixJvo6", 2020-11-09T18:15:51.456 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.456 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.456 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.456 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.456 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.457 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945592.291372_105.1", 2020-11-09T18:15:51.457 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.457 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-19:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.19", 2020-11-09T18:15:51.457 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:12.291372646Z", 2020-11-09T18:15:51.457 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.457 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.458 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_E6pXupKMFQLC75ImCoCkV74", 2020-11-09T18:15:51.458 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.458 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.458 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_E6pXupKMFQLC75ImCoCkV74", 2020-11-09T18:15:51.458 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.459 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.459 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.459 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.459 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.459 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945592.300193_106.1", 2020-11-09T18:15:51.459 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.460 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-19:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.19", 2020-11-09T18:15:51.460 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:12.300193728Z", 2020-11-09T18:15:51.460 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.460 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.460 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_E6pXupKMFQLC75ImCoCkV74", 2020-11-09T18:15:51.460 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.461 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.461 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_E6pXupKMFQLC75ImCoCkV74", 2020-11-09T18:15:51.461 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.461 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.461 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.461 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.462 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.462 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.462 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945594.039798_107.1", 2020-11-09T18:15:51.462 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.462 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-20:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.20", 2020-11-09T18:15:51.462 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:14.039798004Z", 2020-11-09T18:15:51.462 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.463 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.463 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.463 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.463 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.463 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_cfJ_IuSbXauFncnrlMQUrK1", 2020-11-09T18:15:51.463 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.464 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.464 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.464 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.464 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.464 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.464 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945594.043998_109.1", 2020-11-09T18:15:51.465 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.465 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-20:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.20", 2020-11-09T18:15:51.465 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:14.043998666Z", 2020-11-09T18:15:51.465 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.465 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.465 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.465 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.466 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.466 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.466 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_cfJ_IuSbXauFncnrlMQUrK1", 2020-11-09T18:15:51.466 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.466 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.466 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.466 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.467 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.467 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.467 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.467 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945594.046065_110.1", 2020-11-09T18:15:51.467 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.467 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-20", 2020-11-09T18:15:51.467 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:14.046065820Z", 2020-11-09T18:15:51.468 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.468 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.468 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.468 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.468 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.468 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_4-ey3kn91fmGef8EsodcNMf", 2020-11-09T18:15:51.469 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.469 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.469 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.469 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.469 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.469 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.469 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.469 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.470 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945594.050612_111.1", 2020-11-09T18:15:51.470 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.470 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-20", 2020-11-09T18:15:51.470 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:14.050612141Z", 2020-11-09T18:15:51.470 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.470 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.470 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.471 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.471 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.471 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_4-ey3kn91fmGef8EsodcNMf", 2020-11-09T18:15:51.471 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.471 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.472 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.472 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.472 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.472 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.472 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.472 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945595.459954_112.1", 2020-11-09T18:15:51.472 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.472 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-21:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.21", 2020-11-09T18:15:51.473 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:15.459954913Z", 2020-11-09T18:15:51.473 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.473 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.473 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.473 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.473 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.474 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_7weq2-_FEkT51PfhS1TopF2", 2020-11-09T18:15:51.474 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.474 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.474 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.474 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.474 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.475 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.475 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945595.464408_114.1", 2020-11-09T18:15:51.475 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.475 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-21:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.21", 2020-11-09T18:15:51.475 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:15.464408755Z", 2020-11-09T18:15:51.475 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.475 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.476 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.476 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.476 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.476 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_7weq2-_FEkT51PfhS1TopF2", 2020-11-09T18:15:51.476 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.477 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.477 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.477 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.477 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.477 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.477 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945595.466475_115.1", 2020-11-09T18:15:51.477 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.478 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-21", 2020-11-09T18:15:51.478 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:15.466475089Z", 2020-11-09T18:15:51.478 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.478 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.478 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.478 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.479 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.479 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_cxcDXnujWtafpwp1_Zf7_7E", 2020-11-09T18:15:51.479 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.479 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.479 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.479 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.480 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.480 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.480 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945595.470860_116.1", 2020-11-09T18:15:51.480 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.480 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-21", 2020-11-09T18:15:51.480 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:15.470860202Z", 2020-11-09T18:15:51.480 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.481 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.481 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.481 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.481 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.481 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_cxcDXnujWtafpwp1_Zf7_7E", 2020-11-09T18:15:51.481 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.482 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.482 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.482 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.482 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.482 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.483 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945595.574082_117.1", 2020-11-09T18:15:51.483 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.483 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-22:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.22", 2020-11-09T18:15:51.483 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:15.574082160Z", 2020-11-09T18:15:51.483 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.483 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.483 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.483 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.484 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.484 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.484 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_gu325NeoTQ-n6-N4RZO_Ufq", 2020-11-09T18:15:51.484 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.484 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.484 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.484 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.485 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.485 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.485 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.485 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945595.578263_118.1", 2020-11-09T18:15:51.485 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.485 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-22:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.22", 2020-11-09T18:15:51.485 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:15.578263069Z", 2020-11-09T18:15:51.486 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.486 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.486 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.486 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.486 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.486 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_gu325NeoTQ-n6-N4RZO_Ufq", 2020-11-09T18:15:51.487 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.487 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.487 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.487 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.487 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.488 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.488 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945595.580706_119.1", 2020-11-09T18:15:51.488 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.488 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-22", 2020-11-09T18:15:51.488 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:15.580706536Z", 2020-11-09T18:15:51.488 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.488 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.488 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.489 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.489 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.489 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_lHWYslSbF-coDhm_0XUYDqP", 2020-11-09T18:15:51.489 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.489 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.490 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.490 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.490 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.490 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.490 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.490 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945595.585376_120.1", 2020-11-09T18:15:51.490 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.491 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-22", 2020-11-09T18:15:51.491 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:15.585376092Z", 2020-11-09T18:15:51.491 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.491 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.491 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.491 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.491 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.492 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_lHWYslSbF-coDhm_0XUYDqP", 2020-11-09T18:15:51.492 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.492 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.492 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.492 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.492 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.493 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.493 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945595.651287_121.1", 2020-11-09T18:15:51.493 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.493 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-23:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.23", 2020-11-09T18:15:51.493 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:15.651287758Z", 2020-11-09T18:15:51.493 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.493 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.494 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.494 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.494 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.494 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.494 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_k8tGHuX9HMmJcr5VgC0V0M4", 2020-11-09T18:15:51.494 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.494 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.495 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.495 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.495 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.495 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.495 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945595.655554_122.1", 2020-11-09T18:15:51.496 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.496 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-23:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.23", 2020-11-09T18:15:51.496 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:15.655554732Z", 2020-11-09T18:15:51.496 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.496 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.496 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.496 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.497 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.497 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_k8tGHuX9HMmJcr5VgC0V0M4", 2020-11-09T18:15:51.497 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.497 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.497 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.497 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.498 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.498 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.498 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.498 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945595.657606_123.1", 2020-11-09T18:15:51.498 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.498 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-23", 2020-11-09T18:15:51.498 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:15.657606349Z", 2020-11-09T18:15:51.499 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.499 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.499 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.499 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.499 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.499 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_rDxg2fTzKxNRTnXPuogbkec", 2020-11-09T18:15:51.499 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.500 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.500 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.500 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.500 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.501 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945595.662266_124.1", 2020-11-09T18:15:51.501 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.501 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-23", 2020-11-09T18:15:51.501 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:15.662266951Z", 2020-11-09T18:15:51.501 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.501 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.501 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.501 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.502 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.502 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_rDxg2fTzKxNRTnXPuogbkec", 2020-11-09T18:15:51.502 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.502 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.502 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.503 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.503 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.503 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945595.694579_125.1", 2020-11-09T18:15:51.503 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.503 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-21:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.21", 2020-11-09T18:15:51.503 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:15.694579560Z", 2020-11-09T18:15:51.504 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.504 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.504 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_7weq2-_FEkT51PfhS1TopF2", 2020-11-09T18:15:51.504 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.504 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.504 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_7weq2-_FEkT51PfhS1TopF2", 2020-11-09T18:15:51.505 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.505 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.505 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.505 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.505 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.506 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945595.699216_126.1", 2020-11-09T18:15:51.506 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.506 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-21:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.21", 2020-11-09T18:15:51.506 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:15.699216478Z", 2020-11-09T18:15:51.506 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.506 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.506 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_7weq2-_FEkT51PfhS1TopF2", 2020-11-09T18:15:51.507 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.507 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.507 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_7weq2-_FEkT51PfhS1TopF2", 2020-11-09T18:15:51.507 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.507 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.508 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.508 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.508 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.508 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945595.704101_127.1", 2020-11-09T18:15:51.508 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.508 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-22:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.22", 2020-11-09T18:15:51.509 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:15.704101323Z", 2020-11-09T18:15:51.509 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.509 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.509 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_gu325NeoTQ-n6-N4RZO_Ufq", 2020-11-09T18:15:51.509 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.509 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.510 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_gu325NeoTQ-n6-N4RZO_Ufq", 2020-11-09T18:15:51.510 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.510 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.510 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.510 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.510 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.511 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945595.709327_128.1", 2020-11-09T18:15:51.511 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.511 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-22:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.22", 2020-11-09T18:15:51.511 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:15.709327694Z", 2020-11-09T18:15:51.511 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.511 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.512 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_gu325NeoTQ-n6-N4RZO_Ufq", 2020-11-09T18:15:51.512 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.512 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.512 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_gu325NeoTQ-n6-N4RZO_Ufq", 2020-11-09T18:15:51.512 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.512 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.513 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.513 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.513 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.513 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.513 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945595.722945_129.1", 2020-11-09T18:15:51.513 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.514 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-23:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.23", 2020-11-09T18:15:51.514 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:15.722945874Z", 2020-11-09T18:15:51.514 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.514 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.514 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_k8tGHuX9HMmJcr5VgC0V0M4", 2020-11-09T18:15:51.514 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.514 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.515 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_k8tGHuX9HMmJcr5VgC0V0M4", 2020-11-09T18:15:51.515 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.515 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.515 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.515 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.515 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.516 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945595.729371_130.1", 2020-11-09T18:15:51.516 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.516 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-23:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.23", 2020-11-09T18:15:51.516 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:15.729371675Z", 2020-11-09T18:15:51.516 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.516 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.517 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_k8tGHuX9HMmJcr5VgC0V0M4", 2020-11-09T18:15:51.517 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.517 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.517 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_k8tGHuX9HMmJcr5VgC0V0M4", 2020-11-09T18:15:51.517 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.518 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.518 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.518 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.518 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.518 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945597.242843_131.1", 2020-11-09T18:15:51.519 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.519 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-24:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.24", 2020-11-09T18:15:51.519 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:17.242843380Z", 2020-11-09T18:15:51.519 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.519 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.519 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.519 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.520 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.520 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_ZWaAaUyrIgJWu47a8QLknCB", 2020-11-09T18:15:51.520 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.520 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.520 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.520 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.521 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.521 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945597.247361_133.1", 2020-11-09T18:15:51.521 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.521 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-24:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.24", 2020-11-09T18:15:51.521 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:17.247361813Z", 2020-11-09T18:15:51.522 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.522 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.522 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.522 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.522 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.522 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_ZWaAaUyrIgJWu47a8QLknCB", 2020-11-09T18:15:51.523 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.523 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.523 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.523 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.523 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.524 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945597.249391_134.1", 2020-11-09T18:15:51.524 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.524 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-24", 2020-11-09T18:15:51.524 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:17.249391359Z", 2020-11-09T18:15:51.524 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.524 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.524 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.525 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.525 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.525 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_M6yzvodjSwrGlU0XXU23tmw", 2020-11-09T18:15:51.525 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.525 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.525 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.526 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.526 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.526 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945597.257906_135.1", 2020-11-09T18:15:51.526 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.526 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-24", 2020-11-09T18:15:51.527 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:17.257906751Z", 2020-11-09T18:15:51.527 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.527 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.527 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.527 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.527 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.527 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_M6yzvodjSwrGlU0XXU23tmw", 2020-11-09T18:15:51.528 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.528 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.528 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.528 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.528 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.529 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945598.021538_136.1", 2020-11-09T18:15:51.529 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.529 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-25:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.25", 2020-11-09T18:15:51.529 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:18.021538398Z", 2020-11-09T18:15:51.529 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.529 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.529 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.530 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.530 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.530 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dm5hw1Gq9w9JdDaVBV0h96_", 2020-11-09T18:15:51.530 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.530 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.531 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.531 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.531 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.531 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945598.026407_137.1", 2020-11-09T18:15:51.531 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.532 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-25:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.25", 2020-11-09T18:15:51.532 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:18.026407438Z", 2020-11-09T18:15:51.532 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.532 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.532 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.532 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.533 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.533 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dm5hw1Gq9w9JdDaVBV0h96_", 2020-11-09T18:15:51.533 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.533 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.533 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.533 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.534 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.534 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945598.028359_138.1", 2020-11-09T18:15:51.534 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.534 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-25", 2020-11-09T18:15:51.534 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:18.028359151Z", 2020-11-09T18:15:51.535 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.535 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.535 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.535 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.535 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.535 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_hJWxzyq2VXkrZe0lTr6UJ0X", 2020-11-09T18:15:51.535 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.536 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.536 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.536 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.536 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.537 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945598.032307_139.1", 2020-11-09T18:15:51.537 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.537 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-25", 2020-11-09T18:15:51.537 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:18.032307526Z", 2020-11-09T18:15:51.537 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.537 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.537 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.538 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.538 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.538 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_hJWxzyq2VXkrZe0lTr6UJ0X", 2020-11-09T18:15:51.538 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.538 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.539 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.539 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.539 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.539 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945598.347441_140.1", 2020-11-09T18:15:51.539 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.540 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-26:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.26", 2020-11-09T18:15:51.540 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:18.347441566Z", 2020-11-09T18:15:51.540 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.540 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.540 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.541 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.541 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.541 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_tUH5PREzBpkbjt0vmvKAgOc", 2020-11-09T18:15:51.541 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.541 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.542 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.542 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.542 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.542 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945598.351465_141.1", 2020-11-09T18:15:51.542 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.542 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-26:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.26", 2020-11-09T18:15:51.543 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:18.351465217Z", 2020-11-09T18:15:51.543 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.543 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.543 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.543 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.543 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.544 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_tUH5PREzBpkbjt0vmvKAgOc", 2020-11-09T18:15:51.544 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.544 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.544 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.544 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.544 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.545 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945598.353325_142.1", 2020-11-09T18:15:51.545 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.545 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-26", 2020-11-09T18:15:51.545 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:18.353325221Z", 2020-11-09T18:15:51.545 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.545 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.546 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.546 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.546 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.546 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_Z6tqFl2xPji_I9drTzIP2KZ", 2020-11-09T18:15:51.546 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.547 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.547 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.547 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.547 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.547 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945598.357392_143.1", 2020-11-09T18:15:51.547 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.548 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-26", 2020-11-09T18:15:51.548 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:18.357392493Z", 2020-11-09T18:15:51.548 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.548 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.548 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.548 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.549 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.549 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_Z6tqFl2xPji_I9drTzIP2KZ", 2020-11-09T18:15:51.549 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.549 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.549 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.549 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.549 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.550 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945598.450979_144.1", 2020-11-09T18:15:51.550 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.550 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-24:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.24", 2020-11-09T18:15:51.550 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:18.450979415Z", 2020-11-09T18:15:51.550 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.551 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.551 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_ZWaAaUyrIgJWu47a8QLknCB", 2020-11-09T18:15:51.551 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.551 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.551 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_ZWaAaUyrIgJWu47a8QLknCB", 2020-11-09T18:15:51.551 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.552 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.552 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.552 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.552 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.552 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945598.461497_145.1", 2020-11-09T18:15:51.553 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.553 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-24:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.24", 2020-11-09T18:15:51.553 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:18.461497100Z", 2020-11-09T18:15:51.553 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.553 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.553 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_ZWaAaUyrIgJWu47a8QLknCB", 2020-11-09T18:15:51.553 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.554 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.554 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_ZWaAaUyrIgJWu47a8QLknCB", 2020-11-09T18:15:51.554 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.554 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.554 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.554 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.555 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.555 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945598.470055_146.1", 2020-11-09T18:15:51.555 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.555 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-24:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.24", 2020-11-09T18:15:51.555 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:18.470055992Z", 2020-11-09T18:15:51.556 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.556 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.556 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_ZWaAaUyrIgJWu47a8QLknCB", 2020-11-09T18:15:51.556 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.557 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.557 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_ZWaAaUyrIgJWu47a8QLknCB", 2020-11-09T18:15:51.557 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 3 2020-11-09T18:15:51.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.557 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.557 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.558 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.558 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.558 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945598.478206_147.1", 2020-11-09T18:15:51.558 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.558 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-24:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.24", 2020-11-09T18:15:51.558 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:18.478206910Z", 2020-11-09T18:15:51.559 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.559 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.559 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_ZWaAaUyrIgJWu47a8QLknCB", 2020-11-09T18:15:51.559 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.559 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.559 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_ZWaAaUyrIgJWu47a8QLknCB", 2020-11-09T18:15:51.560 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 3 2020-11-09T18:15:51.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.560 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.560 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.560 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.560 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.561 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945598.482582_148.1", 2020-11-09T18:15:51.561 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.561 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-25:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.25", 2020-11-09T18:15:51.561 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:18.482582905Z", 2020-11-09T18:15:51.561 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.561 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.562 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dm5hw1Gq9w9JdDaVBV0h96_", 2020-11-09T18:15:51.562 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.562 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.562 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dm5hw1Gq9w9JdDaVBV0h96_", 2020-11-09T18:15:51.562 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.562 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.563 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.563 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.563 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.563 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945598.488285_149.1", 2020-11-09T18:15:51.563 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.564 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-25:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.25", 2020-11-09T18:15:51.564 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:18.488285331Z", 2020-11-09T18:15:51.564 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.564 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.564 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dm5hw1Gq9w9JdDaVBV0h96_", 2020-11-09T18:15:51.564 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.565 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.565 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dm5hw1Gq9w9JdDaVBV0h96_", 2020-11-09T18:15:51.565 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.565 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.565 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.565 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.565 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.566 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945598.495708_150.1", 2020-11-09T18:15:51.566 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.566 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-25:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.25", 2020-11-09T18:15:51.566 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:18.495708983Z", 2020-11-09T18:15:51.566 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.567 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.567 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dm5hw1Gq9w9JdDaVBV0h96_", 2020-11-09T18:15:51.567 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.567 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.567 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dm5hw1Gq9w9JdDaVBV0h96_", 2020-11-09T18:15:51.567 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 3 2020-11-09T18:15:51.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.568 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.568 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.568 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.568 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.568 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945598.502185_151.1", 2020-11-09T18:15:51.569 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.569 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-25:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.25", 2020-11-09T18:15:51.569 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:18.502185684Z", 2020-11-09T18:15:51.569 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.569 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.569 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dm5hw1Gq9w9JdDaVBV0h96_", 2020-11-09T18:15:51.569 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.570 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.570 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dm5hw1Gq9w9JdDaVBV0h96_", 2020-11-09T18:15:51.570 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 3 2020-11-09T18:15:51.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.570 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.570 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.571 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.571 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.571 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945598.514968_152.1", 2020-11-09T18:15:51.571 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.571 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-26:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.26", 2020-11-09T18:15:51.571 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:18.514968069Z", 2020-11-09T18:15:51.572 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.572 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.572 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_tUH5PREzBpkbjt0vmvKAgOc", 2020-11-09T18:15:51.572 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.572 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.572 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_tUH5PREzBpkbjt0vmvKAgOc", 2020-11-09T18:15:51.573 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.573 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.573 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.573 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.573 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.574 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945598.528310_153.1", 2020-11-09T18:15:51.574 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.574 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-26:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.26", 2020-11-09T18:15:51.574 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:18.528310207Z", 2020-11-09T18:15:51.574 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.574 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.574 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_tUH5PREzBpkbjt0vmvKAgOc", 2020-11-09T18:15:51.575 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.575 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.575 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_tUH5PREzBpkbjt0vmvKAgOc", 2020-11-09T18:15:51.575 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.575 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.576 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.576 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.576 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.576 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945598.543001_154.1", 2020-11-09T18:15:51.576 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.577 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-26:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.26", 2020-11-09T18:15:51.577 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:18.543001147Z", 2020-11-09T18:15:51.577 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.577 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.577 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_tUH5PREzBpkbjt0vmvKAgOc", 2020-11-09T18:15:51.577 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.577 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.578 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_tUH5PREzBpkbjt0vmvKAgOc", 2020-11-09T18:15:51.578 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 3 2020-11-09T18:15:51.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.578 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.578 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.578 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.578 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.579 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945598.548940_155.1", 2020-11-09T18:15:51.579 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.579 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-26:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.26", 2020-11-09T18:15:51.579 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:18.548940287Z", 2020-11-09T18:15:51.579 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.579 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.580 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_tUH5PREzBpkbjt0vmvKAgOc", 2020-11-09T18:15:51.580 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.580 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.580 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_tUH5PREzBpkbjt0vmvKAgOc", 2020-11-09T18:15:51.580 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 3 2020-11-09T18:15:51.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.581 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.581 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.581 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.581 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.581 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945600.502084_156.1", 2020-11-09T18:15:51.581 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.582 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-27:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.27", 2020-11-09T18:15:51.582 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:20.502084837Z", 2020-11-09T18:15:51.582 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.582 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.582 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.582 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.583 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.583 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_ffjKv4uq3V4IpykJfSlfwhB", 2020-11-09T18:15:51.583 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.583 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.584 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.584 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.584 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.584 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945600.507516_158.1", 2020-11-09T18:15:51.585 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.585 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-27:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.27", 2020-11-09T18:15:51.585 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:20.507516916Z", 2020-11-09T18:15:51.585 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.585 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.585 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.586 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.586 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.586 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_ffjKv4uq3V4IpykJfSlfwhB", 2020-11-09T18:15:51.586 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.586 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.587 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.587 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.587 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.587 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945600.509661_159.1", 2020-11-09T18:15:51.587 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.588 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-27", 2020-11-09T18:15:51.588 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:20.509661301Z", 2020-11-09T18:15:51.588 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.588 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.588 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.588 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.588 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.589 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_UJQ25D20xb-Xxms2LevItvh", 2020-11-09T18:15:51.589 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.589 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.589 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.589 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.589 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.590 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945600.513944_160.1", 2020-11-09T18:15:51.590 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.590 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-27", 2020-11-09T18:15:51.590 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:20.513944844Z", 2020-11-09T18:15:51.590 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.590 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.591 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.591 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.591 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.591 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_UJQ25D20xb-Xxms2LevItvh", 2020-11-09T18:15:51.591 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.592 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.592 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.592 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.592 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.592 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945601.434311_161.1", 2020-11-09T18:15:51.592 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.593 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-28:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.28", 2020-11-09T18:15:51.593 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:21.434311607Z", 2020-11-09T18:15:51.593 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.593 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.593 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.593 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.594 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.594 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_jFhIAKXmQdsa5R34lXgk3pv", 2020-11-09T18:15:51.594 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.594 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.594 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.594 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.594 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.595 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945601.438688_162.1", 2020-11-09T18:15:51.595 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.595 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-28:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.28", 2020-11-09T18:15:51.595 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:21.438688009Z", 2020-11-09T18:15:51.595 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.596 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.596 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.596 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.596 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.596 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_jFhIAKXmQdsa5R34lXgk3pv", 2020-11-09T18:15:51.596 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.597 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.597 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.597 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.597 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.597 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945601.440791_163.1", 2020-11-09T18:15:51.598 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.598 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-28", 2020-11-09T18:15:51.598 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:21.440791127Z", 2020-11-09T18:15:51.598 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.598 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.598 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.598 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.599 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.599 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_4xBDSjdegdCsfVupUBmzzRX", 2020-11-09T18:15:51.599 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.599 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.599 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.599 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.600 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.600 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.600 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945601.445125_164.1", 2020-11-09T18:15:51.600 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.600 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-28", 2020-11-09T18:15:51.600 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:21.445125336Z", 2020-11-09T18:15:51.601 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.601 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.601 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.601 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.601 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.601 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_4xBDSjdegdCsfVupUBmzzRX", 2020-11-09T18:15:51.602 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.602 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.602 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.602 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.602 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.603 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.603 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945603.023496_165.1", 2020-11-09T18:15:51.603 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.603 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-29:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.29", 2020-11-09T18:15:51.603 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:23.023496753Z", 2020-11-09T18:15:51.603 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.603 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.603 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.604 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.604 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.604 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_P83ahmV5nZ3tNcs0_lSHbDU", 2020-11-09T18:15:51.604 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.604 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.605 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.605 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.605 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.605 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945603.027781_166.1", 2020-11-09T18:15:51.605 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.605 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-29:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.29", 2020-11-09T18:15:51.606 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:23.027781495Z", 2020-11-09T18:15:51.606 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.606 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.606 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.606 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.606 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.607 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_P83ahmV5nZ3tNcs0_lSHbDU", 2020-11-09T18:15:51.607 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.607 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.607 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.607 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.607 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.608 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.608 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945603.030309_167.1", 2020-11-09T18:15:51.608 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.608 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-29", 2020-11-09T18:15:51.608 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:23.030309393Z", 2020-11-09T18:15:51.608 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.608 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.609 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.609 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.609 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.609 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_Hhh1A6pFRNzrIunHlFDZwos", 2020-11-09T18:15:51.609 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.610 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.610 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.610 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.610 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.610 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945603.034891_168.1", 2020-11-09T18:15:51.610 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.611 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-29", 2020-11-09T18:15:51.611 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:23.034891461Z", 2020-11-09T18:15:51.611 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.611 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.611 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.611 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.612 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.612 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_Hhh1A6pFRNzrIunHlFDZwos", 2020-11-09T18:15:51.612 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.612 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.612 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.612 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.612 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.613 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945603.213259_169.1", 2020-11-09T18:15:51.613 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.613 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-27:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.27", 2020-11-09T18:15:51.613 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:23.213259709Z", 2020-11-09T18:15:51.613 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.614 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.614 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_ffjKv4uq3V4IpykJfSlfwhB", 2020-11-09T18:15:51.614 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.614 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.614 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_ffjKv4uq3V4IpykJfSlfwhB", 2020-11-09T18:15:51.614 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.615 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.615 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.615 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.615 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.615 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.615 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945603.223295_170.1", 2020-11-09T18:15:51.616 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.616 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-27:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.27", 2020-11-09T18:15:51.616 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:23.223295591Z", 2020-11-09T18:15:51.616 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.616 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.616 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_ffjKv4uq3V4IpykJfSlfwhB", 2020-11-09T18:15:51.616 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.617 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.617 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_ffjKv4uq3V4IpykJfSlfwhB", 2020-11-09T18:15:51.617 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.617 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.617 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.618 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.618 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.618 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945604.018547_171.1", 2020-11-09T18:15:51.618 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.618 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-28:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.28", 2020-11-09T18:15:51.618 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:24.018547219Z", 2020-11-09T18:15:51.619 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.619 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.619 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_jFhIAKXmQdsa5R34lXgk3pv", 2020-11-09T18:15:51.619 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.619 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.619 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_jFhIAKXmQdsa5R34lXgk3pv", 2020-11-09T18:15:51.620 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.620 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.620 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.620 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.620 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.621 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945604.029428_172.1", 2020-11-09T18:15:51.621 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.621 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-28:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.28", 2020-11-09T18:15:51.621 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:24.029428921Z", 2020-11-09T18:15:51.621 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.621 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.621 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_jFhIAKXmQdsa5R34lXgk3pv", 2020-11-09T18:15:51.622 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.622 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.622 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_jFhIAKXmQdsa5R34lXgk3pv", 2020-11-09T18:15:51.622 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.622 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.623 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.623 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.623 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.623 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945604.720659_173.1", 2020-11-09T18:15:51.623 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.624 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-29:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.29", 2020-11-09T18:15:51.624 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:24.720659097Z", 2020-11-09T18:15:51.624 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.624 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.624 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_P83ahmV5nZ3tNcs0_lSHbDU", 2020-11-09T18:15:51.624 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.624 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.625 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_P83ahmV5nZ3tNcs0_lSHbDU", 2020-11-09T18:15:51.625 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.625 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.625 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.625 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.626 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.626 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.626 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945604.732717_174.1", 2020-11-09T18:15:51.626 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.626 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-29:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.29", 2020-11-09T18:15:51.626 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:24.732717989Z", 2020-11-09T18:15:51.627 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.627 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.627 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_P83ahmV5nZ3tNcs0_lSHbDU", 2020-11-09T18:15:51.627 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.627 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.627 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_P83ahmV5nZ3tNcs0_lSHbDU", 2020-11-09T18:15:51.627 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.628 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.628 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.628 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.628 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.629 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945605.406938_175.1", 2020-11-09T18:15:51.629 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.629 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-30:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.30", 2020-11-09T18:15:51.629 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:25.406938541Z", 2020-11-09T18:15:51.629 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.629 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.629 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.630 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.630 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.630 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_2k6PaixusxbzBO-jYw6D0Md", 2020-11-09T18:15:51.630 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.630 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.631 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.631 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.631 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.631 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945605.412027_176.1", 2020-11-09T18:15:51.631 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.631 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-30:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.30", 2020-11-09T18:15:51.632 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:25.412027440Z", 2020-11-09T18:15:51.632 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.632 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.632 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.632 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.632 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.633 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_2k6PaixusxbzBO-jYw6D0Md", 2020-11-09T18:15:51.633 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.633 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.633 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.633 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.633 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.634 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.634 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945605.413981_177.1", 2020-11-09T18:15:51.634 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.634 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-30", 2020-11-09T18:15:51.634 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:25.413981551Z", 2020-11-09T18:15:51.634 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.634 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.635 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.635 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.635 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.635 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_OVN11Cxzp_HRsHiBhSltQ2E", 2020-11-09T18:15:51.635 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.635 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.636 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.636 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.636 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.636 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945605.419171_178.1", 2020-11-09T18:15:51.636 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.637 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-30", 2020-11-09T18:15:51.637 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:25.419171989Z", 2020-11-09T18:15:51.637 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.637 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.637 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.637 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.637 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.638 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_OVN11Cxzp_HRsHiBhSltQ2E", 2020-11-09T18:15:51.638 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.638 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.638 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.638 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.638 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.639 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945605.813895_179.1", 2020-11-09T18:15:51.639 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.639 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-31:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.31", 2020-11-09T18:15:51.639 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:25.813895626Z", 2020-11-09T18:15:51.640 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.640 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.640 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.640 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.640 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.640 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dboGSnUFvBxtlhGOt6TYcuV", 2020-11-09T18:15:51.641 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.641 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.641 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.641 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.641 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.642 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.642 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945605.819203_180.1", 2020-11-09T18:15:51.642 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.642 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-31:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.31", 2020-11-09T18:15:51.642 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:25.819203440Z", 2020-11-09T18:15:51.642 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.643 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.643 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.643 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.643 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.643 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dboGSnUFvBxtlhGOt6TYcuV", 2020-11-09T18:15:51.643 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.644 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.644 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.644 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.644 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.644 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.644 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945605.821244_181.1", 2020-11-09T18:15:51.645 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.645 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-31", 2020-11-09T18:15:51.645 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:25.821244081Z", 2020-11-09T18:15:51.645 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.645 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.645 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.645 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.646 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.646 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_D_zBqUuf_PUMl6KPscj9tcq", 2020-11-09T18:15:51.646 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.646 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.646 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.647 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.647 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.647 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945605.825854_182.1", 2020-11-09T18:15:51.647 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.647 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-31", 2020-11-09T18:15:51.647 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:25.825854764Z", 2020-11-09T18:15:51.648 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.648 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.648 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.648 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.648 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.648 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_D_zBqUuf_PUMl6KPscj9tcq", 2020-11-09T18:15:51.649 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.649 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.649 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.649 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.649 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.650 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.314889_183.1", 2020-11-09T18:15:51.650 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.650 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-32:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.32", 2020-11-09T18:15:51.650 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.314889240Z", 2020-11-09T18:15:51.650 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.650 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.650 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.651 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.651 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.651 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_h7iYndUz6ZZeJDRnCAJhq6b", 2020-11-09T18:15:51.651 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.651 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.651 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.652 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.652 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.652 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.319791_184.1", 2020-11-09T18:15:51.652 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.652 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-32:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.32", 2020-11-09T18:15:51.653 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.319791421Z", 2020-11-09T18:15:51.653 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.653 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.653 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.653 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.653 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.653 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_h7iYndUz6ZZeJDRnCAJhq6b", 2020-11-09T18:15:51.654 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.654 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.654 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.654 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.654 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.655 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.321799_185.1", 2020-11-09T18:15:51.655 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.655 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-32", 2020-11-09T18:15:51.655 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.321799710Z", 2020-11-09T18:15:51.655 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.655 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.656 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.656 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.656 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.656 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_kfgLDq_uLGT_Ii4AzNlTUJd", 2020-11-09T18:15:51.656 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.657 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.657 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.657 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.657 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.657 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.326129_186.1", 2020-11-09T18:15:51.658 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.658 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-32", 2020-11-09T18:15:51.658 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.326129812Z", 2020-11-09T18:15:51.658 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.658 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.658 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.658 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.659 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.659 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_kfgLDq_uLGT_Ii4AzNlTUJd", 2020-11-09T18:15:51.659 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.659 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.659 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.659 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.660 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.660 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.351846_187.1", 2020-11-09T18:15:51.660 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.660 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-30:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.30", 2020-11-09T18:15:51.660 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.351846735Z", 2020-11-09T18:15:51.661 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.661 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.661 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_2k6PaixusxbzBO-jYw6D0Md", 2020-11-09T18:15:51.661 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.661 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.661 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_2k6PaixusxbzBO-jYw6D0Md", 2020-11-09T18:15:51.661 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.662 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.662 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.662 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.662 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.663 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.358962_188.1", 2020-11-09T18:15:51.663 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.663 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-30:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.30", 2020-11-09T18:15:51.663 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.358962403Z", 2020-11-09T18:15:51.663 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.663 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.663 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_2k6PaixusxbzBO-jYw6D0Md", 2020-11-09T18:15:51.664 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.664 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.664 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_2k6PaixusxbzBO-jYw6D0Md", 2020-11-09T18:15:51.664 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.664 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.664 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.665 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.665 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.665 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.413199_189.1", 2020-11-09T18:15:51.665 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.665 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-31:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.31", 2020-11-09T18:15:51.666 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.413199050Z", 2020-11-09T18:15:51.666 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.666 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.666 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dboGSnUFvBxtlhGOt6TYcuV", 2020-11-09T18:15:51.666 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.666 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.667 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dboGSnUFvBxtlhGOt6TYcuV", 2020-11-09T18:15:51.667 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.667 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.667 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.667 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.667 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.668 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.423298_190.1", 2020-11-09T18:15:51.668 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.668 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-31:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.31", 2020-11-09T18:15:51.668 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.423298203Z", 2020-11-09T18:15:51.668 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.668 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dboGSnUFvBxtlhGOt6TYcuV", 2020-11-09T18:15:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dboGSnUFvBxtlhGOt6TYcuV", 2020-11-09T18:15:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.670 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.670 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.670 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.670 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.670 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.486367_191.1", 2020-11-09T18:15:51.670 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.671 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-32:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.32", 2020-11-09T18:15:51.671 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.486367158Z", 2020-11-09T18:15:51.671 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.671 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.671 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_h7iYndUz6ZZeJDRnCAJhq6b", 2020-11-09T18:15:51.671 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.671 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.672 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_h7iYndUz6ZZeJDRnCAJhq6b", 2020-11-09T18:15:51.672 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.672 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.672 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.672 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.672 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.673 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.497483_192.1", 2020-11-09T18:15:51.673 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.673 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-32:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.32", 2020-11-09T18:15:51.673 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.497483092Z", 2020-11-09T18:15:51.673 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.673 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.674 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_h7iYndUz6ZZeJDRnCAJhq6b", 2020-11-09T18:15:51.674 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.674 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.674 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_h7iYndUz6ZZeJDRnCAJhq6b", 2020-11-09T18:15:51.674 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.675 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.675 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.675 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.675 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.675 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.567936_193.1", 2020-11-09T18:15:51.676 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.676 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-33:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.33", 2020-11-09T18:15:51.676 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.567936942Z", 2020-11-09T18:15:51.676 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.676 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.676 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.676 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dV7Zwb-EMDhOHgweLJlLD79", 2020-11-09T18:15:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.678 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.678 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.678 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.573179_194.1", 2020-11-09T18:15:51.678 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.678 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-33:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.33", 2020-11-09T18:15:51.678 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.573179777Z", 2020-11-09T18:15:51.678 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.679 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.679 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.679 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.679 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.679 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dV7Zwb-EMDhOHgweLJlLD79", 2020-11-09T18:15:51.679 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.575137_195.1", 2020-11-09T18:15:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-33", 2020-11-09T18:15:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.575137143Z", 2020-11-09T18:15:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.682 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.682 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "__9pytw2itoqGRYgrLwyaE0v", 2020-11-09T18:15:51.682 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.682 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.682 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.578921_196.1", 2020-11-09T18:15:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-33", 2020-11-09T18:15:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.578921064Z", 2020-11-09T18:15:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "__9pytw2itoqGRYgrLwyaE0v", 2020-11-09T18:15:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.686 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.603950_198.1", 2020-11-09T18:15:51.686 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.686 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-34:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.34", 2020-11-09T18:15:51.686 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.603950696Z", 2020-11-09T18:15:51.686 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.686 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.687 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.687 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.687 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.687 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_KazzTgKF9mHRl9t-mjqUZIF", 2020-11-09T18:15:51.687 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.687 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.608737_199.1", 2020-11-09T18:15:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.689 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-34:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.34", 2020-11-09T18:15:51.689 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.608737440Z", 2020-11-09T18:15:51.689 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.689 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.689 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.689 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.689 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.690 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_KazzTgKF9mHRl9t-mjqUZIF", 2020-11-09T18:15:51.690 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.690 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.690 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.690 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.690 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.610541_200.1", 2020-11-09T18:15:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-34", 2020-11-09T18:15:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.610541395Z", 2020-11-09T18:15:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_QkHzIdBeCUUuKSc9kaJ3NT5", 2020-11-09T18:15:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.693 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.693 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.693 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.693 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.693 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.614352_201.1", 2020-11-09T18:15:51.694 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.694 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-34", 2020-11-09T18:15:51.694 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.614352129Z", 2020-11-09T18:15:51.694 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.694 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.694 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.694 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.695 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.695 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_QkHzIdBeCUUuKSc9kaJ3NT5", 2020-11-09T18:15:51.695 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.695 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.695 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.695 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.696 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.696 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.658773_202.1", 2020-11-09T18:15:51.696 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.696 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-35:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.35", 2020-11-09T18:15:51.696 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.658773780Z", 2020-11-09T18:15:51.696 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.697 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.697 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.697 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.697 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.697 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dAiojqzewqe1JboY85aFSyJ", 2020-11-09T18:15:51.697 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.698 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.698 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.698 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.698 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.699 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.663192_203.1", 2020-11-09T18:15:51.699 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.699 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-35:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.35", 2020-11-09T18:15:51.699 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.663192679Z", 2020-11-09T18:15:51.699 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.699 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.699 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.699 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.700 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.700 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dAiojqzewqe1JboY85aFSyJ", 2020-11-09T18:15:51.700 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.700 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.700 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.701 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.701 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.701 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.665284_204.1", 2020-11-09T18:15:51.701 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.701 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-35", 2020-11-09T18:15:51.701 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.665284756Z", 2020-11-09T18:15:51.702 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.702 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.702 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.702 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.702 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.702 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_YIG4M8hwsEikvpJIsO0cCiX", 2020-11-09T18:15:51.703 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.703 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.703 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.703 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.703 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.704 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.671390_205.1", 2020-11-09T18:15:51.704 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.704 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-35", 2020-11-09T18:15:51.704 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.671390803Z", 2020-11-09T18:15:51.704 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.704 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.705 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.705 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.705 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.705 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_YIG4M8hwsEikvpJIsO0cCiX", 2020-11-09T18:15:51.705 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.705 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.706 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.706 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.706 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.706 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.708610_206.1", 2020-11-09T18:15:51.706 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.707 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-33:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.33", 2020-11-09T18:15:51.707 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.708610357Z", 2020-11-09T18:15:51.707 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.707 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.707 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dV7Zwb-EMDhOHgweLJlLD79", 2020-11-09T18:15:51.707 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.708 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.708 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dV7Zwb-EMDhOHgweLJlLD79", 2020-11-09T18:15:51.708 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.708 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.708 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.708 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.708 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.709 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.713328_207.1", 2020-11-09T18:15:51.709 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.709 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-33:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.33", 2020-11-09T18:15:51.709 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.713328296Z", 2020-11-09T18:15:51.709 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.710 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.710 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dV7Zwb-EMDhOHgweLJlLD79", 2020-11-09T18:15:51.710 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.710 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.710 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dV7Zwb-EMDhOHgweLJlLD79", 2020-11-09T18:15:51.710 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.711 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.711 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.711 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.711 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.711 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945607.760169_210.1", 2020-11-09T18:15:51.712 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.712 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-34:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.34", 2020-11-09T18:15:51.712 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:27.760169945Z", 2020-11-09T18:15:51.712 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.712 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.712 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_KazzTgKF9mHRl9t-mjqUZIF", 2020-11-09T18:15:51.712 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.713 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.713 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_KazzTgKF9mHRl9t-mjqUZIF", 2020-11-09T18:15:51.713 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.713 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.713 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.713 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.714 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.714 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945607.771847_211.1", 2020-11-09T18:15:51.714 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.714 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-34:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.34", 2020-11-09T18:15:51.714 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:27.771847568Z", 2020-11-09T18:15:51.714 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.715 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.715 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_KazzTgKF9mHRl9t-mjqUZIF", 2020-11-09T18:15:51.715 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.715 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.715 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_KazzTgKF9mHRl9t-mjqUZIF", 2020-11-09T18:15:51.715 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.716 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.716 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.716 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.716 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.716 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.717 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945609.277502_212.1", 2020-11-09T18:15:51.717 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.717 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-35:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.35", 2020-11-09T18:15:51.717 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:29.277502241Z", 2020-11-09T18:15:51.717 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.717 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.717 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dAiojqzewqe1JboY85aFSyJ", 2020-11-09T18:15:51.717 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.718 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.718 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dAiojqzewqe1JboY85aFSyJ", 2020-11-09T18:15:51.718 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.718 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.718 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.719 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.719 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.719 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945609.288552_213.1", 2020-11-09T18:15:51.719 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.719 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-35:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.35", 2020-11-09T18:15:51.720 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:29.288552016Z", 2020-11-09T18:15:51.720 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.720 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.720 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dAiojqzewqe1JboY85aFSyJ", 2020-11-09T18:15:51.720 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.720 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.720 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dAiojqzewqe1JboY85aFSyJ", 2020-11-09T18:15:51.721 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.721 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.721 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.721 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.721 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.722 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.722 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945609.660249_214.1", 2020-11-09T18:15:51.722 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.722 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-36:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.36", 2020-11-09T18:15:51.722 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:29.660249505Z", 2020-11-09T18:15:51.722 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.722 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.723 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.723 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.723 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.723 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_ozJuqjR3GVh8aGnB2Xh4hR5", 2020-11-09T18:15:51.723 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.723 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.724 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.724 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.724 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.724 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945609.665942_215.1", 2020-11-09T18:15:51.724 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.725 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-36:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.36", 2020-11-09T18:15:51.725 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:29.665942367Z", 2020-11-09T18:15:51.725 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.725 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.725 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.725 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.725 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.726 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_ozJuqjR3GVh8aGnB2Xh4hR5", 2020-11-09T18:15:51.726 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.726 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.726 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.726 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.726 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.727 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.727 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945609.669229_216.1", 2020-11-09T18:15:51.727 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.727 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-36", 2020-11-09T18:15:51.727 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:29.669229416Z", 2020-11-09T18:15:51.727 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.728 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.728 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.728 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.728 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.728 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_DZ1KbRgq_ERertkZGzu4RA_", 2020-11-09T18:15:51.728 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.729 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.729 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.729 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.729 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.729 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945609.675301_217.1", 2020-11-09T18:15:51.730 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.730 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-36", 2020-11-09T18:15:51.730 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:29.675301279Z", 2020-11-09T18:15:51.730 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.730 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.730 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.730 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.731 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.731 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_DZ1KbRgq_ERertkZGzu4RA_", 2020-11-09T18:15:51.731 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.731 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.731 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.731 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.732 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.732 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945610.079342_218.1", 2020-11-09T18:15:51.732 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.732 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-37:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.37", 2020-11-09T18:15:51.732 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:30.079342479Z", 2020-11-09T18:15:51.732 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.733 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.733 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.733 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.733 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.733 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_fx1DINtpIECPzOGaSfs3ucL", 2020-11-09T18:15:51.733 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.734 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.734 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.734 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.734 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.734 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.735 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945610.083364_219.1", 2020-11-09T18:15:51.735 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.735 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-37:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.37", 2020-11-09T18:15:51.735 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:30.083364883Z", 2020-11-09T18:15:51.735 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.735 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.735 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.735 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.736 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.736 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_fx1DINtpIECPzOGaSfs3ucL", 2020-11-09T18:15:51.736 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.736 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.736 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.737 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.737 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.737 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.737 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945610.085275_220.1", 2020-11-09T18:15:51.737 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.737 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-37", 2020-11-09T18:15:51.737 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:30.085275759Z", 2020-11-09T18:15:51.738 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.738 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.738 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.738 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.738 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.738 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_J2nfKVSwwp-9vo2WCGxBAhy", 2020-11-09T18:15:51.739 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.739 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.739 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.739 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.739 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.739 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.740 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945610.091583_221.1", 2020-11-09T18:15:51.740 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.740 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-37", 2020-11-09T18:15:51.740 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:30.091583370Z", 2020-11-09T18:15:51.740 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.740 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.740 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.741 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.741 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.741 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_J2nfKVSwwp-9vo2WCGxBAhy", 2020-11-09T18:15:51.741 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.741 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.741 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.742 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.742 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.742 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.742 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.742 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945610.137385_222.1", 2020-11-09T18:15:51.742 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.742 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-38:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.38", 2020-11-09T18:15:51.743 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:30.137385505Z", 2020-11-09T18:15:51.743 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.743 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.743 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.743 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.743 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.743 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_MKdJSqznasfgUfD8pSqNyMN", 2020-11-09T18:15:51.744 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.744 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.744 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.744 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.744 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.745 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.745 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945610.141970_223.1", 2020-11-09T18:15:51.745 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.745 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-38:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.38", 2020-11-09T18:15:51.745 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:30.141970597Z", 2020-11-09T18:15:51.745 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.745 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.746 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.746 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.746 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.746 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.746 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_MKdJSqznasfgUfD8pSqNyMN", 2020-11-09T18:15:51.746 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.746 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.746 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.747 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.747 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.747 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.747 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945610.144161_224.1", 2020-11-09T18:15:51.747 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.748 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-38", 2020-11-09T18:15:51.748 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:30.144161714Z", 2020-11-09T18:15:51.748 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.748 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.748 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.748 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.748 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.749 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_Lsx6lFhkZQ0sbnB7SgpHP7X", 2020-11-09T18:15:51.749 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.749 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.749 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.749 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.749 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.750 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.750 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945610.148613_225.1", 2020-11-09T18:15:51.750 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.750 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-38", 2020-11-09T18:15:51.750 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:30.148613440Z", 2020-11-09T18:15:51.750 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.751 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.751 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.751 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.751 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.751 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.751 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_Lsx6lFhkZQ0sbnB7SgpHP7X", 2020-11-09T18:15:51.751 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.751 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.752 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.752 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.752 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.752 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.752 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.752 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945611.763559_226.1", 2020-11-09T18:15:51.752 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.753 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-39:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.39", 2020-11-09T18:15:51.753 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:31.763559546Z", 2020-11-09T18:15:51.753 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.753 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.753 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.753 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.754 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.754 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_DoCr53MUjfmeA1PTWsLAoZg", 2020-11-09T18:15:51.754 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.754 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.754 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.754 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.754 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.755 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.755 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.755 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945611.768768_228.1", 2020-11-09T18:15:51.755 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.755 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-39:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.39", 2020-11-09T18:15:51.755 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:31.768768717Z", 2020-11-09T18:15:51.755 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.756 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.756 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.756 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.756 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.756 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.756 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_DoCr53MUjfmeA1PTWsLAoZg", 2020-11-09T18:15:51.756 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.757 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.757 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.757 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.757 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.757 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.757 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.757 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.757 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945611.771007_229.1", 2020-11-09T18:15:51.758 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.758 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-39", 2020-11-09T18:15:51.758 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:31.771007696Z", 2020-11-09T18:15:51.758 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.758 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.758 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.758 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.759 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.759 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.759 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_pGgwgGXPCFeV9dl7j-mkG2D", 2020-11-09T18:15:51.759 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.759 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.759 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.759 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.759 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.760 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.760 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.760 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945611.776767_230.1", 2020-11-09T18:15:51.760 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.760 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-39", 2020-11-09T18:15:51.760 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:31.776767187Z", 2020-11-09T18:15:51.761 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.761 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.761 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.761 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.761 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.761 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.761 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_pGgwgGXPCFeV9dl7j-mkG2D", 2020-11-09T18:15:51.761 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.762 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.762 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.762 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.762 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.762 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.762 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.762 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.763 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945612.536576_231.1", 2020-11-09T18:15:51.763 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.763 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-40:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.40", 2020-11-09T18:15:51.763 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:32.536576361Z", 2020-11-09T18:15:51.763 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.763 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.763 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.764 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.764 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.764 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.764 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_eRiB6BdOLC_U7ajfFCuPyzq", 2020-11-09T18:15:51.764 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.764 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.764 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.764 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.765 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.765 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.765 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.765 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945612.542749_232.1", 2020-11-09T18:15:51.765 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.765 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-40:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.40", 2020-11-09T18:15:51.766 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:32.542749842Z", 2020-11-09T18:15:51.766 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.766 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.766 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.766 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.766 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.767 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_eRiB6BdOLC_U7ajfFCuPyzq", 2020-11-09T18:15:51.767 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.767 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.767 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.767 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.767 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.768 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.768 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945612.545398_233.1", 2020-11-09T18:15:51.768 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.768 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-40", 2020-11-09T18:15:51.768 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:32.545398453Z", 2020-11-09T18:15:51.768 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.768 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.769 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.769 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.769 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.769 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_v-PHMVkQGnPvIMVgQXzEv-e", 2020-11-09T18:15:51.769 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.769 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.770 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.770 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.770 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.770 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.770 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.770 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945612.550287_234.1", 2020-11-09T18:15:51.770 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.771 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-40", 2020-11-09T18:15:51.771 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:32.550287369Z", 2020-11-09T18:15:51.771 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.771 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.771 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.771 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.771 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.772 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.772 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_v-PHMVkQGnPvIMVgQXzEv-e", 2020-11-09T18:15:51.772 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.772 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.772 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.772 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.772 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.772 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.773 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.773 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945612.641557_236.1", 2020-11-09T18:15:51.773 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.774 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-41:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.41", 2020-11-09T18:15:51.775 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:32.641557244Z", 2020-11-09T18:15:51.775 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.775 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.775 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.775 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.775 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.776 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_VbEzzl6PqzRKUgwPXZGxMNK", 2020-11-09T18:15:51.776 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.776 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.776 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.776 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.776 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.776 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.777 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.777 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945612.646933_237.1", 2020-11-09T18:15:51.777 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.777 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-41:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.41", 2020-11-09T18:15:51.777 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:32.646933508Z", 2020-11-09T18:15:51.777 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.778 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.778 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.778 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.778 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.778 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_VbEzzl6PqzRKUgwPXZGxMNK", 2020-11-09T18:15:51.778 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.779 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.779 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.779 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.779 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.779 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945612.649001_238.1", 2020-11-09T18:15:51.780 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.780 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-41", 2020-11-09T18:15:51.780 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:32.649001607Z", 2020-11-09T18:15:51.780 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.780 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.780 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.780 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.781 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.781 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_-QlJbrN7eIOTl-coxNoWB0_", 2020-11-09T18:15:51.781 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.781 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.781 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.781 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.782 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.782 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.782 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.782 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945612.653005_239.1", 2020-11-09T18:15:51.782 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.782 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-41", 2020-11-09T18:15:51.782 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:32.653005302Z", 2020-11-09T18:15:51.783 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.783 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.783 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.783 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.783 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.783 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_-QlJbrN7eIOTl-coxNoWB0_", 2020-11-09T18:15:51.784 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.784 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.784 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.784 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.784 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.785 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945563.557787_47.1", 2020-11-09T18:15:51.785 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.785 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-1:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.1", 2020-11-09T18:15:51.785 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:43.557787352Z", 2020-11-09T18:15:51.785 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.785 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.785 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.786 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.786 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.786 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_5-8ThY0rD5nJEgi4q-i6qW8", 2020-11-09T18:15:51.786 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.786 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.787 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.787 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.787 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.787 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.787 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945563.567624_51.1", 2020-11-09T18:15:51.787 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.788 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-1:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.1", 2020-11-09T18:15:51.788 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:43.567624280Z", 2020-11-09T18:15:51.788 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.788 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.788 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.788 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.788 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.788 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.789 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_5-8ThY0rD5nJEgi4q-i6qW8", 2020-11-09T18:15:51.789 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.789 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.789 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.789 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.789 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.790 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.790 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945563.569310_52.1", 2020-11-09T18:15:51.790 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.790 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-1", 2020-11-09T18:15:51.790 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:43.569310797Z", 2020-11-09T18:15:51.790 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.791 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.791 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.791 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.791 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.791 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_KrRBCosiXlBz0ajusgkrpWS", 2020-11-09T18:15:51.791 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.792 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.792 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.792 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.792 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.792 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.792 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945563.577890_53.1", 2020-11-09T18:15:51.793 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.793 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-1", 2020-11-09T18:15:51.793 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:43.577890725Z", 2020-11-09T18:15:51.793 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.793 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.793 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.793 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.794 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.794 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_KrRBCosiXlBz0ajusgkrpWS", 2020-11-09T18:15:51.794 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.794 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.794 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.794 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.795 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.795 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945563.608973_54.1", 2020-11-09T18:15:51.795 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.795 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-2:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.2", 2020-11-09T18:15:51.795 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:43.608973238Z", 2020-11-09T18:15:51.795 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.796 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.796 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.796 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.796 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.796 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_eZ3F6MzV7ES6qOIMaFzRrOO", 2020-11-09T18:15:51.796 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.797 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.797 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.797 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.797 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.797 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.797 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945563.618816_55.1", 2020-11-09T18:15:51.798 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.798 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-2:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.2", 2020-11-09T18:15:51.798 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:43.618816140Z", 2020-11-09T18:15:51.798 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.798 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.798 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.798 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.799 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.799 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_eZ3F6MzV7ES6qOIMaFzRrOO", 2020-11-09T18:15:51.799 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.799 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.799 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.799 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.800 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.800 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945563.621200_56.1", 2020-11-09T18:15:51.800 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.800 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-2", 2020-11-09T18:15:51.800 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:43.621200541Z", 2020-11-09T18:15:51.801 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.801 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.801 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.801 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.801 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.801 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_rj9KSIrvtNvXYWU2UYD1Z_m", 2020-11-09T18:15:51.802 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.802 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.802 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.802 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.802 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.803 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945563.627302_57.1", 2020-11-09T18:15:51.803 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.803 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-2", 2020-11-09T18:15:51.803 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:43.627302782Z", 2020-11-09T18:15:51.803 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.803 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.803 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.804 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.804 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.804 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_rj9KSIrvtNvXYWU2UYD1Z_m", 2020-11-09T18:15:51.804 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.804 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.804 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.805 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.805 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.805 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945567.627267_58.1", 2020-11-09T18:15:51.805 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.805 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-3:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.3", 2020-11-09T18:15:51.806 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:47.627267860Z", 2020-11-09T18:15:51.806 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.806 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.806 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.806 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.806 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.807 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_TnwqP2_mu3KZnFJOXhnO-DD", 2020-11-09T18:15:51.807 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.807 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.807 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.807 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.807 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.808 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945567.630834_59.1", 2020-11-09T18:15:51.808 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.808 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-3:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.3", 2020-11-09T18:15:51.808 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:47.630834733Z", 2020-11-09T18:15:51.808 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.808 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.809 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.809 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.809 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.809 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_TnwqP2_mu3KZnFJOXhnO-DD", 2020-11-09T18:15:51.809 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.809 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.810 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.810 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.810 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.810 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945567.632589_60.1", 2020-11-09T18:15:51.810 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.811 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-3", 2020-11-09T18:15:51.811 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:47.632589277Z", 2020-11-09T18:15:51.811 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.811 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.811 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.811 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.811 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.812 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_MHzcuL8U9yY5bsA8iZvWhTc", 2020-11-09T18:15:51.812 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.812 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.812 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.812 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.812 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.813 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945567.636634_61.1", 2020-11-09T18:15:51.813 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.813 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-3", 2020-11-09T18:15:51.813 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:47.636634661Z", 2020-11-09T18:15:51.813 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.814 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.814 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.814 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.814 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.814 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_MHzcuL8U9yY5bsA8iZvWhTc", 2020-11-09T18:15:51.814 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.815 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.815 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.815 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.815 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.815 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945570.370676_62.1", 2020-11-09T18:15:51.816 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.816 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-4:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.4", 2020-11-09T18:15:51.816 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:50.370676662Z", 2020-11-09T18:15:51.816 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.816 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.816 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.816 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.817 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.817 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_ufcC505t7ROkSzsmsL1irho", 2020-11-09T18:15:51.817 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.817 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.817 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.817 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.818 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.818 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945570.375714_64.1", 2020-11-09T18:15:51.818 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.818 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-4:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.4", 2020-11-09T18:15:51.818 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:50.375714519Z", 2020-11-09T18:15:51.819 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.819 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.819 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.819 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.819 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.819 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_ufcC505t7ROkSzsmsL1irho", 2020-11-09T18:15:51.819 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.820 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.820 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.820 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.820 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.821 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945570.377850_65.1", 2020-11-09T18:15:51.821 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.821 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-4", 2020-11-09T18:15:51.821 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:50.377850670Z", 2020-11-09T18:15:51.821 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.821 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.821 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.821 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.822 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.822 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_xnR_YIJ8zUxxn61yz_DTUkk", 2020-11-09T18:15:51.822 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.822 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.822 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.823 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.823 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.823 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945570.382459_66.1", 2020-11-09T18:15:51.823 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.823 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-4", 2020-11-09T18:15:51.824 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:50.382459732Z", 2020-11-09T18:15:51.824 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.824 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.824 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.824 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.824 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.824 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_xnR_YIJ8zUxxn61yz_DTUkk", 2020-11-09T18:15:51.825 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.825 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.825 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.825 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.825 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.826 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945570.406070_67.1", 2020-11-09T18:15:51.826 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.826 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-5:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.5", 2020-11-09T18:15:51.826 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:50.406070925Z", 2020-11-09T18:15:51.826 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.826 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.826 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.827 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.827 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.827 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_UShZGGBB4kBaxRzXtVmZBUW", 2020-11-09T18:15:51.827 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.827 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.827 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.828 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.828 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.828 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945570.410945_68.1", 2020-11-09T18:15:51.828 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.828 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-5:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.5", 2020-11-09T18:15:51.829 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:50.410945255Z", 2020-11-09T18:15:51.829 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.829 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.829 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.829 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.829 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.829 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_UShZGGBB4kBaxRzXtVmZBUW", 2020-11-09T18:15:51.830 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.830 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.830 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.830 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.830 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.831 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945570.412796_69.1", 2020-11-09T18:15:51.831 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.831 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-5", 2020-11-09T18:15:51.831 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:50.412796913Z", 2020-11-09T18:15:51.831 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.831 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.831 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.832 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.832 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.832 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_5bx-BH8ijLijK0Hb5tRXntZ", 2020-11-09T18:15:51.832 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.832 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.833 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.833 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.833 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.833 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945570.416832_70.1", 2020-11-09T18:15:51.833 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.833 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-5", 2020-11-09T18:15:51.834 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:50.416832729Z", 2020-11-09T18:15:51.834 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.834 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.834 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.834 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.834 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.835 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_5bx-BH8ijLijK0Hb5tRXntZ", 2020-11-09T18:15:51.835 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.835 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.835 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.835 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.835 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.836 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945570.461105_71.1", 2020-11-09T18:15:51.836 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.836 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-6:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.6", 2020-11-09T18:15:51.836 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:50.461105755Z", 2020-11-09T18:15:51.836 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.836 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.837 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.837 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.837 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.837 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_qWlHsdsWYJHc0mQkDC3vzYx", 2020-11-09T18:15:51.837 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.838 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.838 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.838 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.838 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.838 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945570.473537_72.1", 2020-11-09T18:15:51.838 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.839 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-6:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.6", 2020-11-09T18:15:51.839 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:50.473537938Z", 2020-11-09T18:15:51.839 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.839 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.839 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.839 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.840 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.840 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_qWlHsdsWYJHc0mQkDC3vzYx", 2020-11-09T18:15:51.840 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.840 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.840 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.840 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.840 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.841 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945570.477860_73.1", 2020-11-09T18:15:51.841 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.841 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-6", 2020-11-09T18:15:51.841 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:50.477860682Z", 2020-11-09T18:15:51.841 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.842 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.842 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.842 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.842 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.842 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dmuyyeGa1O--zFbpSa0SA8D", 2020-11-09T18:15:51.842 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.843 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.843 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.843 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.843 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.843 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945570.489122_74.1", 2020-11-09T18:15:51.844 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.844 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-6", 2020-11-09T18:15:51.844 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:50.489122994Z", 2020-11-09T18:15:51.844 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.844 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.844 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.844 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.845 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.845 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dmuyyeGa1O--zFbpSa0SA8D", 2020-11-09T18:15:51.845 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.845 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.845 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.845 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.846 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.846 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945579.353942_75.1", 2020-11-09T18:15:51.846 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.846 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-7:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.7", 2020-11-09T18:15:51.846 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:59.353942489Z", 2020-11-09T18:15:51.847 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.847 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.847 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.847 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.847 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.847 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_GIkCDw3GWmDRO3-A0auRtxB", 2020-11-09T18:15:51.847 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.848 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.848 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.848 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.848 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.848 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945579.359041_78.1", 2020-11-09T18:15:51.849 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.849 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-7:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.7", 2020-11-09T18:15:51.849 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:59.359041450Z", 2020-11-09T18:15:51.849 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.849 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.849 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.849 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.850 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.850 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_GIkCDw3GWmDRO3-A0auRtxB", 2020-11-09T18:15:51.850 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.850 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.850 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.850 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.851 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.851 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945579.361526_79.1", 2020-11-09T18:15:51.851 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.851 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-7", 2020-11-09T18:15:51.851 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:59.361526900Z", 2020-11-09T18:15:51.852 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.852 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.852 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.852 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.852 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.852 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_euurhSSx6WmV693QUVvezHc", 2020-11-09T18:15:51.852 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.853 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.853 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.853 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.853 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.854 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945579.367322_80.1", 2020-11-09T18:15:51.854 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.854 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-7", 2020-11-09T18:15:51.854 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:59.367322143Z", 2020-11-09T18:15:51.854 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.854 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.854 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.854 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.855 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.855 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_euurhSSx6WmV693QUVvezHc", 2020-11-09T18:15:51.855 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.855 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.855 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.856 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.856 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.856 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945579.395388_81.1", 2020-11-09T18:15:51.856 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.856 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-8:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.8", 2020-11-09T18:15:51.857 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:59.395388143Z", 2020-11-09T18:15:51.857 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.857 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.857 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.857 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.857 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.857 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_yBzYlhlGW2rcaJy91e-7eds", 2020-11-09T18:15:51.858 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.858 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.858 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.858 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.858 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.859 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945579.400026_82.1", 2020-11-09T18:15:51.859 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.859 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-8:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.8", 2020-11-09T18:15:51.859 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:59.400026920Z", 2020-11-09T18:15:51.859 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.859 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.859 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.860 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.860 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.860 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_yBzYlhlGW2rcaJy91e-7eds", 2020-11-09T18:15:51.860 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.860 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.861 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.861 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.861 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.861 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945579.402183_83.1", 2020-11-09T18:15:51.861 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.861 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-8", 2020-11-09T18:15:51.862 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:59.402183037Z", 2020-11-09T18:15:51.862 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.862 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.862 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.862 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.862 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.863 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_OGdV9KW-t2xB2NYjc8pNB2U", 2020-11-09T18:15:51.863 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.863 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.863 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.863 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.863 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.864 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945579.406495_84.1", 2020-11-09T18:15:51.864 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.864 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-8", 2020-11-09T18:15:51.864 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:59.406495328Z", 2020-11-09T18:15:51.864 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.864 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.864 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.865 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.865 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.865 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_OGdV9KW-t2xB2NYjc8pNB2U", 2020-11-09T18:15:51.865 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.865 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.866 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.866 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.866 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.866 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945579.450093_85.1", 2020-11-09T18:15:51.866 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.866 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-9:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.9", 2020-11-09T18:15:51.867 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:59.450093555Z", 2020-11-09T18:15:51.867 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.867 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.867 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.867 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.867 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.868 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_sx-mxXXsFhATfbPuJXuh1It", 2020-11-09T18:15:51.868 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.868 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.868 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.868 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.868 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.869 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945579.455372_86.1", 2020-11-09T18:15:51.869 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:51.869 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-9:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.9", 2020-11-09T18:15:51.869 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:59.455372433Z", 2020-11-09T18:15:51.869 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.869 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.869 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.870 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.870 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.870 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_sx-mxXXsFhATfbPuJXuh1It", 2020-11-09T18:15:51.870 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.870 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.871 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.871 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.871 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.871 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945579.459352_87.1", 2020-11-09T18:15:51.871 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.872 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-9", 2020-11-09T18:15:51.872 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:59.459352980Z", 2020-11-09T18:15:51.872 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.872 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.872 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.872 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.872 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.873 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_CehTzP8PVTogocbVBJZMUE-", 2020-11-09T18:15:51.873 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.873 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.873 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:51.873 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.873 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.874 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945579.467305_88.1", 2020-11-09T18:15:51.874 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.874 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-9", 2020-11-09T18:15:51.874 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:59.467305227Z", 2020-11-09T18:15:51.874 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.874 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.875 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:51.875 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:51.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.875 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.875 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_CehTzP8PVTogocbVBJZMUE-", 2020-11-09T18:15:51.875 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.876 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.876 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.876 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.876 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.876 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945582.240772_89.1", 2020-11-09T18:15:51.876 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.877 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-7", 2020-11-09T18:15:51.877 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.240772877Z", 2020-11-09T18:15:51.877 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.877 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.877 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_euurhSSx6WmV693QUVvezHc", 2020-11-09T18:15:51.877 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.877 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.878 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_euurhSSx6WmV693QUVvezHc", 2020-11-09T18:15:51.878 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.878 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.878 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "remove" 2020-11-09T18:15:51.878 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.878 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.879 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945582.251051_91.1", 2020-11-09T18:15:51.879 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.879 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-7", 2020-11-09T18:15:51.879 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.251051719Z", 2020-11-09T18:15:51.879 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.880 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.880 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_euurhSSx6WmV693QUVvezHc", 2020-11-09T18:15:51.880 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.880 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.880 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_euurhSSx6WmV693QUVvezHc", 2020-11-09T18:15:51.880 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.881 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.881 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.881 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.881 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.881 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945582.313479_92.1", 2020-11-09T18:15:51.881 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.882 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-8", 2020-11-09T18:15:51.882 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.313479625Z", 2020-11-09T18:15:51.882 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.882 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.882 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_OGdV9KW-t2xB2NYjc8pNB2U", 2020-11-09T18:15:51.882 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.883 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.883 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_OGdV9KW-t2xB2NYjc8pNB2U", 2020-11-09T18:15:51.883 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.883 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.883 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "remove" 2020-11-09T18:15:51.883 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.883 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.884 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945582.321374_93.1", 2020-11-09T18:15:51.884 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.884 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-8", 2020-11-09T18:15:51.884 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.321374298Z", 2020-11-09T18:15:51.884 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.885 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.885 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_OGdV9KW-t2xB2NYjc8pNB2U", 2020-11-09T18:15:51.885 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.885 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.885 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_OGdV9KW-t2xB2NYjc8pNB2U", 2020-11-09T18:15:51.885 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.886 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.886 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.886 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.886 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.886 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945582.425686_94.1", 2020-11-09T18:15:51.887 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.887 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-9", 2020-11-09T18:15:51.887 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.425686180Z", 2020-11-09T18:15:51.887 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.887 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.887 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_CehTzP8PVTogocbVBJZMUE-", 2020-11-09T18:15:51.887 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.888 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.888 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_CehTzP8PVTogocbVBJZMUE-", 2020-11-09T18:15:51.888 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.888 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.888 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "remove" 2020-11-09T18:15:51.888 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.889 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:51.889 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945582.432939_95.1", 2020-11-09T18:15:51.889 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:51.889 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-9", 2020-11-09T18:15:51.889 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939430Z", 2020-11-09T18:15:51.889 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:51.890 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:51.890 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_CehTzP8PVTogocbVBJZMUE-", 2020-11-09T18:15:51.890 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:51.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.890 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:51.890 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_CehTzP8PVTogocbVBJZMUE-", 2020-11-09T18:15:51.890 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:51.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:51.891 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:51.891 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:51.891 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.891 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.891 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:51.891 INFO:teuthology.orchestra.run.smithi078.stdout:] 2020-11-09T18:15:51.893 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:51.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:51.894 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:15:51.934 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:15:51.934 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:15:52.176 INFO:teuthology.orchestra.run.smithi051.stdout:[ 2020-11-09T18:15:52.177 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.177 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945439.484659_48.1", 2020-11-09T18:15:52.178 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "user", 2020-11-09T18:15:52.178 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "rgw-multisite-test-user", 2020-11-09T18:15:52.178 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:10:39.484659067Z", 2020-11-09T18:15:52.179 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.179 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.179 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.179 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.179 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.180 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.180 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_qxUmJfyn2gLxVOOHvE_t0VQ", 2020-11-09T18:15:52.180 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.180 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.181 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.181 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.181 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.182 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.182 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.182 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.183 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945439.492888_50.1", 2020-11-09T18:15:52.183 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "user", 2020-11-09T18:15:52.183 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "rgw-multisite-test-user", 2020-11-09T18:15:52.184 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:10:39.492888346Z", 2020-11-09T18:15:52.184 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.184 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.184 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.185 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.185 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.185 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.186 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_qxUmJfyn2gLxVOOHvE_t0VQ", 2020-11-09T18:15:52.186 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.186 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.186 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.187 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.187 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.188 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.189 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.189 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.189 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945586.256023_51.1", 2020-11-09T18:15:52.189 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.189 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-10:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.10", 2020-11-09T18:15:52.189 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:06.256023190Z", 2020-11-09T18:15:52.190 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.190 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.190 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.190 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.190 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.190 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.191 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_xk1WELKUHZIqmUd5oTGWrEo", 2020-11-09T18:15:52.191 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.191 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.191 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.191 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.191 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.191 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.191 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.192 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.192 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945586.261184_59.1", 2020-11-09T18:15:52.192 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.192 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-10:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.10", 2020-11-09T18:15:52.192 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:06.261184780Z", 2020-11-09T18:15:52.192 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.192 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.193 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.193 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.193 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.193 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.193 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_xk1WELKUHZIqmUd5oTGWrEo", 2020-11-09T18:15:52.193 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.193 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.194 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.194 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.194 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.194 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.194 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.194 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.194 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945586.263844_60.1", 2020-11-09T18:15:52.195 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.195 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-10", 2020-11-09T18:15:52.195 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:06.263844660Z", 2020-11-09T18:15:52.195 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.195 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.195 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.195 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.196 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.196 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.196 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_p-7-bX-sO2hh9J_2Nonu8rp", 2020-11-09T18:15:52.196 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.196 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.196 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.196 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.196 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.197 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.197 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.197 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.197 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945586.268581_61.1", 2020-11-09T18:15:52.197 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.197 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-10", 2020-11-09T18:15:52.197 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:06.268581555Z", 2020-11-09T18:15:52.198 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.198 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.198 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.198 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.198 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.198 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.198 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_p-7-bX-sO2hh9J_2Nonu8rp", 2020-11-09T18:15:52.199 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.199 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.199 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.199 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.199 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.199 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.199 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.200 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.200 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945587.202762_62.1", 2020-11-09T18:15:52.200 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.200 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-11:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.11", 2020-11-09T18:15:52.200 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:07.202762446Z", 2020-11-09T18:15:52.200 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.200 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.200 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.201 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.201 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.201 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.201 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_5opGpz-RxCA9co32z-sLP0e", 2020-11-09T18:15:52.201 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.201 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.201 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.202 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.202 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.202 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.202 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.202 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.202 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945587.207905_63.1", 2020-11-09T18:15:52.202 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.203 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-11:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.11", 2020-11-09T18:15:52.203 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:07.207905728Z", 2020-11-09T18:15:52.203 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.203 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.203 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.203 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.203 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.203 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.204 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_5opGpz-RxCA9co32z-sLP0e", 2020-11-09T18:15:52.204 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.204 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.204 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.204 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.204 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.204 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.205 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.205 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.205 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945587.210891_64.1", 2020-11-09T18:15:52.205 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.205 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-11", 2020-11-09T18:15:52.205 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:07.210891570Z", 2020-11-09T18:15:52.205 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.205 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.206 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.206 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.206 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.206 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.206 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_QiNkPq8a5F1hoeuAtqCxHNQ", 2020-11-09T18:15:52.206 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.206 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.207 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.207 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.207 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.207 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.207 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.207 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.207 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945587.216871_65.1", 2020-11-09T18:15:52.207 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.208 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-11", 2020-11-09T18:15:52.208 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:07.216871948Z", 2020-11-09T18:15:52.208 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.208 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.208 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.208 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.209 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.209 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.209 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_QiNkPq8a5F1hoeuAtqCxHNQ", 2020-11-09T18:15:52.209 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.209 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.210 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.210 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.210 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.210 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.210 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.210 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.210 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945588.126275_66.1", 2020-11-09T18:15:52.211 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.211 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-12:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.12", 2020-11-09T18:15:52.211 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:08.126275141Z", 2020-11-09T18:15:52.211 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.211 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.211 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.211 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.211 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.212 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.212 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_7WQjdW6VYC3fP3T1DUBlsCk", 2020-11-09T18:15:52.212 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.212 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.212 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.212 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.212 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.213 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.213 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.213 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.213 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945588.133515_67.1", 2020-11-09T18:15:52.213 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.213 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-12:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.12", 2020-11-09T18:15:52.213 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:08.133515560Z", 2020-11-09T18:15:52.214 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.214 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.214 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.214 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.214 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.214 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.214 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_7WQjdW6VYC3fP3T1DUBlsCk", 2020-11-09T18:15:52.215 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.215 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.215 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.215 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.215 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.215 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.215 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.215 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.216 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945588.135642_68.1", 2020-11-09T18:15:52.216 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.216 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-12", 2020-11-09T18:15:52.216 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:08.135642294Z", 2020-11-09T18:15:52.216 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.216 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.216 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.217 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.217 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.217 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.217 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_PdQ7asc_0k_3_XotzK7eatQ", 2020-11-09T18:15:52.217 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.217 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.217 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.218 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.218 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.218 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.218 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.218 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.218 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945588.143746_69.1", 2020-11-09T18:15:52.219 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.219 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-12", 2020-11-09T18:15:52.219 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:08.143746572Z", 2020-11-09T18:15:52.219 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.219 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.219 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.219 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.220 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.220 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.220 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_PdQ7asc_0k_3_XotzK7eatQ", 2020-11-09T18:15:52.220 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.220 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.220 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.220 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.221 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.221 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.221 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.221 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.221 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945588.381357_70.1", 2020-11-09T18:15:52.221 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.221 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-13:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.13", 2020-11-09T18:15:52.221 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:08.381357113Z", 2020-11-09T18:15:52.222 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.222 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.222 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.222 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.223 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.223 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.223 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_jRsLDHCkxygm_NRmusaemxh", 2020-11-09T18:15:52.223 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.223 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.223 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.223 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.224 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.224 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.224 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.224 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.224 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945588.386292_71.1", 2020-11-09T18:15:52.224 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.225 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-13:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.13", 2020-11-09T18:15:52.225 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:08.386292596Z", 2020-11-09T18:15:52.225 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.225 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.225 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.225 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.226 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.226 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.226 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_jRsLDHCkxygm_NRmusaemxh", 2020-11-09T18:15:52.226 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.226 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.226 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.227 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.227 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.227 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.227 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.227 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.227 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945588.391790_72.1", 2020-11-09T18:15:52.227 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.227 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-13", 2020-11-09T18:15:52.228 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:08.391790067Z", 2020-11-09T18:15:52.228 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.228 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.229 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.229 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.230 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.230 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.230 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_1zAxS7Eu4GCsgwIw7eCrrnz", 2020-11-09T18:15:52.230 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.230 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.230 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.231 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.231 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.231 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.231 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.231 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.231 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945588.397490_73.1", 2020-11-09T18:15:52.231 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.231 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-13", 2020-11-09T18:15:52.232 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:08.397490757Z", 2020-11-09T18:15:52.232 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.232 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.232 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.232 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.232 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.232 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.233 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_1zAxS7Eu4GCsgwIw7eCrrnz", 2020-11-09T18:15:52.233 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.233 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.233 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.233 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.233 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.233 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.234 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.234 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.234 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945589.380394_74.1", 2020-11-09T18:15:52.234 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.234 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-14:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.14", 2020-11-09T18:15:52.234 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:09.380394647Z", 2020-11-09T18:15:52.235 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.235 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.235 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.235 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.235 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.235 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.236 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_ZoR7I5lOnVtZkMZ3aHO24Vp", 2020-11-09T18:15:52.236 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.236 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.236 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.236 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.236 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.236 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.237 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.237 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.237 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945589.385117_75.1", 2020-11-09T18:15:52.237 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.237 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-14:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.14", 2020-11-09T18:15:52.237 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:09.385117237Z", 2020-11-09T18:15:52.237 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.238 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.238 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.238 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.238 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.238 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.239 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_ZoR7I5lOnVtZkMZ3aHO24Vp", 2020-11-09T18:15:52.239 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.239 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.239 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.239 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.239 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.240 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.240 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.240 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.240 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945589.387155_76.1", 2020-11-09T18:15:52.240 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.240 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-14", 2020-11-09T18:15:52.240 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:09.387155581Z", 2020-11-09T18:15:52.240 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.241 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.241 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.241 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.241 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.241 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.241 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_iQYh1hCLyur_paDmK5MzSiw", 2020-11-09T18:15:52.241 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.242 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.242 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.242 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.242 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.242 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.242 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.242 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.243 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945589.391251_77.1", 2020-11-09T18:15:52.243 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.243 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-14", 2020-11-09T18:15:52.243 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:09.391251540Z", 2020-11-09T18:15:52.243 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.243 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.243 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.244 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.244 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.244 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.244 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_iQYh1hCLyur_paDmK5MzSiw", 2020-11-09T18:15:52.244 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.245 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.245 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.245 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.245 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.246 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.246 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.246 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.247 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945589.523751_78.1", 2020-11-09T18:15:52.247 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.247 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-15:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.15", 2020-11-09T18:15:52.247 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:09.523751440Z", 2020-11-09T18:15:52.248 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.248 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.248 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.249 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.249 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.249 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.249 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_KVNlDu4zRBEGnM1a31uWniG", 2020-11-09T18:15:52.250 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.250 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.250 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.251 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.251 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.251 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.252 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.252 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.253 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945589.534160_79.1", 2020-11-09T18:15:52.253 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.253 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-15:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.15", 2020-11-09T18:15:52.253 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:09.534160171Z", 2020-11-09T18:15:52.254 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.254 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.254 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.255 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.255 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.255 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.256 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_KVNlDu4zRBEGnM1a31uWniG", 2020-11-09T18:15:52.256 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.256 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.256 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.257 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.257 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.257 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.258 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.258 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.258 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945589.536862_80.1", 2020-11-09T18:15:52.258 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.259 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-15", 2020-11-09T18:15:52.259 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:09.536862933Z", 2020-11-09T18:15:52.259 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.260 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.260 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.260 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.260 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.261 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.261 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_Cq6C5mA4ASiylyTIC5SMBlP", 2020-11-09T18:15:52.261 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.262 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.262 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.262 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.262 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.263 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.263 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.263 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.264 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945589.541920_81.1", 2020-11-09T18:15:52.264 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.264 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-15", 2020-11-09T18:15:52.264 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:09.541920734Z", 2020-11-09T18:15:52.265 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.265 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.265 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.266 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.266 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.266 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.267 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_Cq6C5mA4ASiylyTIC5SMBlP", 2020-11-09T18:15:52.267 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.267 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.267 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.268 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.268 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.268 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.268 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.269 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.269 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945590.011767_82.1", 2020-11-09T18:15:52.269 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.270 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-16:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.16", 2020-11-09T18:15:52.270 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:10.011767329Z", 2020-11-09T18:15:52.270 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.271 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.271 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.271 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.271 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.272 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.272 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_fJsd-7NZirDCEodHfdpY9oA", 2020-11-09T18:15:52.272 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.273 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.273 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.273 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.273 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.274 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.274 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.274 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.275 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945590.016722_83.1", 2020-11-09T18:15:52.275 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.275 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-16:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.16", 2020-11-09T18:15:52.276 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:10.016722284Z", 2020-11-09T18:15:52.276 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.276 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.276 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.277 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.277 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.277 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.277 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_fJsd-7NZirDCEodHfdpY9oA", 2020-11-09T18:15:52.278 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.278 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.278 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.279 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.279 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.279 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.279 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.280 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.280 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945590.019556_84.1", 2020-11-09T18:15:52.280 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.281 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-16", 2020-11-09T18:15:52.281 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:10.019556366Z", 2020-11-09T18:15:52.281 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.281 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.282 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.282 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.282 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.283 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.283 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_83VseKNDEzwg2zXJyAxVSNV", 2020-11-09T18:15:52.283 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.284 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.284 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.284 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.284 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.285 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.285 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.285 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.286 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945590.028620_85.1", 2020-11-09T18:15:52.286 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.286 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-16", 2020-11-09T18:15:52.286 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:10.028620030Z", 2020-11-09T18:15:52.287 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.287 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.287 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.288 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.288 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.288 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.289 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_83VseKNDEzwg2zXJyAxVSNV", 2020-11-09T18:15:52.289 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.289 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.289 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.290 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.290 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.290 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.290 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.291 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.291 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945590.059189_86.1", 2020-11-09T18:15:52.291 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.292 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-17:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.17", 2020-11-09T18:15:52.292 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:10.059189312Z", 2020-11-09T18:15:52.292 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.293 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.293 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.293 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.293 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.294 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.294 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_nhpbrE0BaTecbpBGZZxkC_9", 2020-11-09T18:15:52.295 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.295 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.295 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.296 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.296 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.296 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.296 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.296 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.296 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945590.065317_87.1", 2020-11-09T18:15:52.296 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.297 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-17:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.17", 2020-11-09T18:15:52.297 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:10.065317701Z", 2020-11-09T18:15:52.297 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.297 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.297 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.297 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.297 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.298 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.298 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_nhpbrE0BaTecbpBGZZxkC_9", 2020-11-09T18:15:52.298 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.298 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.298 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.298 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.299 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.299 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.299 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.299 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.299 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945590.068336_88.1", 2020-11-09T18:15:52.299 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.300 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-17", 2020-11-09T18:15:52.300 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:10.068336838Z", 2020-11-09T18:15:52.300 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.300 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.300 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.300 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.301 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.301 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.301 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_DZ4ozYh7iBiXpSqfO2MFhvn", 2020-11-09T18:15:52.301 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.301 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.302 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.302 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.302 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.302 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.302 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.302 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.303 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945590.076749_89.1", 2020-11-09T18:15:52.303 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.303 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-17", 2020-11-09T18:15:52.303 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:10.076749917Z", 2020-11-09T18:15:52.303 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.303 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.303 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.303 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.304 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.304 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.304 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_DZ4ozYh7iBiXpSqfO2MFhvn", 2020-11-09T18:15:52.304 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.304 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.304 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.304 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.305 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.305 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.305 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.305 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.305 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945590.120726_90.1", 2020-11-09T18:15:52.305 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.305 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-18:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.18", 2020-11-09T18:15:52.306 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:10.120726234Z", 2020-11-09T18:15:52.306 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.306 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.306 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.306 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.306 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.306 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.307 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_R8b8xLVdGNeTGDdC42EuEfE", 2020-11-09T18:15:52.307 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.307 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.307 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.307 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.307 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.307 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.308 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.308 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.308 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945590.127213_91.1", 2020-11-09T18:15:52.308 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.308 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-18:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.18", 2020-11-09T18:15:52.308 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:10.127213093Z", 2020-11-09T18:15:52.308 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.308 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.309 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.309 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.309 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.309 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.309 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_R8b8xLVdGNeTGDdC42EuEfE", 2020-11-09T18:15:52.309 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.309 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.310 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.310 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.310 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.310 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.310 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.310 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.310 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945590.129116_92.1", 2020-11-09T18:15:52.311 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.311 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-18", 2020-11-09T18:15:52.311 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:10.129116612Z", 2020-11-09T18:15:52.311 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.311 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.311 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.311 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.312 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.312 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.312 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_jvrbBsWxGWFBmAxvo-GL8oI", 2020-11-09T18:15:52.312 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.312 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.312 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.312 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.312 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.313 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.313 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.313 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.313 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945590.133065_93.1", 2020-11-09T18:15:52.313 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.313 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-18", 2020-11-09T18:15:52.313 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:10.133065786Z", 2020-11-09T18:15:52.314 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.314 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.314 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.314 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.314 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.314 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.314 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_jvrbBsWxGWFBmAxvo-GL8oI", 2020-11-09T18:15:52.315 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.315 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.315 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.315 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.315 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.315 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.315 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.316 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.316 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945590.156864_94.1", 2020-11-09T18:15:52.316 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.316 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-16:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.16", 2020-11-09T18:15:52.316 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:10.156864142Z", 2020-11-09T18:15:52.316 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.316 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.317 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_fJsd-7NZirDCEodHfdpY9oA", 2020-11-09T18:15:52.317 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.317 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.317 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.317 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_fJsd-7NZirDCEodHfdpY9oA", 2020-11-09T18:15:52.317 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.317 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.317 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.318 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.318 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.318 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.318 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.318 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.318 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945590.163596_95.1", 2020-11-09T18:15:52.318 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.319 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-16:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.16", 2020-11-09T18:15:52.319 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:10.163596031Z", 2020-11-09T18:15:52.319 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.319 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.319 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_fJsd-7NZirDCEodHfdpY9oA", 2020-11-09T18:15:52.319 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.319 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.320 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.320 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_fJsd-7NZirDCEodHfdpY9oA", 2020-11-09T18:15:52.320 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.320 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.320 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.320 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.320 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.320 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.321 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.321 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.321 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945590.168397_96.1", 2020-11-09T18:15:52.321 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.321 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-17:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.17", 2020-11-09T18:15:52.321 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:10.168397876Z", 2020-11-09T18:15:52.321 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.322 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.322 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_nhpbrE0BaTecbpBGZZxkC_9", 2020-11-09T18:15:52.322 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.322 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.322 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.322 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_nhpbrE0BaTecbpBGZZxkC_9", 2020-11-09T18:15:52.322 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.323 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.323 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.323 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.323 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.323 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.323 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.323 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.324 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945590.178873_97.1", 2020-11-09T18:15:52.324 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.324 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-17:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.17", 2020-11-09T18:15:52.324 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:10.178873937Z", 2020-11-09T18:15:52.324 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.324 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.324 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_nhpbrE0BaTecbpBGZZxkC_9", 2020-11-09T18:15:52.324 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.325 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.325 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.325 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_nhpbrE0BaTecbpBGZZxkC_9", 2020-11-09T18:15:52.325 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.325 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.325 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.325 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.326 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.326 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.326 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.326 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.326 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945590.190493_98.1", 2020-11-09T18:15:52.326 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.326 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-18:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.18", 2020-11-09T18:15:52.327 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:10.190493784Z", 2020-11-09T18:15:52.327 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.327 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.327 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_R8b8xLVdGNeTGDdC42EuEfE", 2020-11-09T18:15:52.327 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.327 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.327 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.328 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_R8b8xLVdGNeTGDdC42EuEfE", 2020-11-09T18:15:52.328 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.328 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.328 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.328 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.328 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.328 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.329 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.329 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.329 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945590.199376_99.1", 2020-11-09T18:15:52.329 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.329 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-18:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.18", 2020-11-09T18:15:52.329 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:10.199376961Z", 2020-11-09T18:15:52.329 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.330 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.330 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_R8b8xLVdGNeTGDdC42EuEfE", 2020-11-09T18:15:52.330 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.330 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.330 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.330 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_R8b8xLVdGNeTGDdC42EuEfE", 2020-11-09T18:15:52.330 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.330 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.331 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.331 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.331 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.331 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.331 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.331 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.331 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945592.270499_100.1", 2020-11-09T18:15:52.332 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.332 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-19:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.19", 2020-11-09T18:15:52.332 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:12.270499729Z", 2020-11-09T18:15:52.332 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.332 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.332 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.332 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.333 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.333 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.333 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_E6pXupKMFQLC75ImCoCkV74", 2020-11-09T18:15:52.333 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.333 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.333 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.333 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.333 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.334 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.334 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.334 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.334 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945592.275423_102.1", 2020-11-09T18:15:52.334 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.334 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-19:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.19", 2020-11-09T18:15:52.335 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:12.275423973Z", 2020-11-09T18:15:52.335 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.335 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.335 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.335 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.335 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.335 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.336 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_E6pXupKMFQLC75ImCoCkV74", 2020-11-09T18:15:52.336 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.336 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.336 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.336 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.336 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.336 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.336 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.337 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.337 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945592.278527_103.1", 2020-11-09T18:15:52.337 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.337 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-19", 2020-11-09T18:15:52.337 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:12.278527677Z", 2020-11-09T18:15:52.337 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.337 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.338 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.338 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.338 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.338 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.338 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_OGzY_hYHGuuTc6_1SixJvo6", 2020-11-09T18:15:52.338 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.338 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.339 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.339 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.339 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.339 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.339 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.339 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.339 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945592.284263_104.1", 2020-11-09T18:15:52.340 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.340 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-19", 2020-11-09T18:15:52.340 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:12.284263022Z", 2020-11-09T18:15:52.340 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.340 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.340 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.340 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.341 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.341 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.341 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_OGzY_hYHGuuTc6_1SixJvo6", 2020-11-09T18:15:52.341 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.341 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.341 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.341 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.341 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.342 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.342 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.342 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.342 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945592.291372_105.1", 2020-11-09T18:15:52.342 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.342 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-19:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.19", 2020-11-09T18:15:52.342 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:12.291372646Z", 2020-11-09T18:15:52.343 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.343 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.343 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_E6pXupKMFQLC75ImCoCkV74", 2020-11-09T18:15:52.343 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.343 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.343 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.344 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_E6pXupKMFQLC75ImCoCkV74", 2020-11-09T18:15:52.344 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.344 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.344 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.344 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.344 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.344 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.345 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.345 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.345 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945592.300193_106.1", 2020-11-09T18:15:52.345 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.345 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-19:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.19", 2020-11-09T18:15:52.345 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:12.300193728Z", 2020-11-09T18:15:52.345 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.346 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.346 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_E6pXupKMFQLC75ImCoCkV74", 2020-11-09T18:15:52.346 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.346 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.346 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.346 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_E6pXupKMFQLC75ImCoCkV74", 2020-11-09T18:15:52.347 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.347 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.347 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.347 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.347 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.347 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.347 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.347 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.348 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945594.039798_107.1", 2020-11-09T18:15:52.348 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.348 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-20:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.20", 2020-11-09T18:15:52.348 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:14.039798004Z", 2020-11-09T18:15:52.348 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.348 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.348 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.349 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.349 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.349 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.349 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_cfJ_IuSbXauFncnrlMQUrK1", 2020-11-09T18:15:52.349 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.349 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.349 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.349 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.350 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.350 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.350 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.350 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.350 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945594.043998_109.1", 2020-11-09T18:15:52.350 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.350 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-20:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.20", 2020-11-09T18:15:52.351 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:14.043998666Z", 2020-11-09T18:15:52.351 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.351 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.351 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.351 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.351 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.352 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.352 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_cfJ_IuSbXauFncnrlMQUrK1", 2020-11-09T18:15:52.352 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.352 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.352 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.352 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.353 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.353 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.353 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.353 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.353 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945594.046065_110.1", 2020-11-09T18:15:52.353 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.353 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-20", 2020-11-09T18:15:52.354 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:14.046065820Z", 2020-11-09T18:15:52.354 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.354 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.354 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.354 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.354 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.354 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.355 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_4-ey3kn91fmGef8EsodcNMf", 2020-11-09T18:15:52.355 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.355 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.355 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.355 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.355 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.355 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.356 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.356 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.356 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945594.050612_111.1", 2020-11-09T18:15:52.356 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.356 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-20", 2020-11-09T18:15:52.356 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:14.050612141Z", 2020-11-09T18:15:52.356 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.356 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.357 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.357 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.357 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.357 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.357 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_4-ey3kn91fmGef8EsodcNMf", 2020-11-09T18:15:52.357 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.357 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.358 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.358 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.358 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.358 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.358 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.358 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.358 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945595.459954_112.1", 2020-11-09T18:15:52.359 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.359 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-21:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.21", 2020-11-09T18:15:52.359 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:15.459954913Z", 2020-11-09T18:15:52.359 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.359 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.359 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.359 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.360 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.360 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.360 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_7weq2-_FEkT51PfhS1TopF2", 2020-11-09T18:15:52.360 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.360 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.360 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.360 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.360 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.361 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.361 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.361 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.361 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945595.464408_114.1", 2020-11-09T18:15:52.361 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.361 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-21:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.21", 2020-11-09T18:15:52.361 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:15.464408755Z", 2020-11-09T18:15:52.362 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.362 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.362 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.362 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.362 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.362 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.362 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_7weq2-_FEkT51PfhS1TopF2", 2020-11-09T18:15:52.363 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.363 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.363 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.363 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.363 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.363 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.363 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.364 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.364 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945595.466475_115.1", 2020-11-09T18:15:52.364 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.364 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-21", 2020-11-09T18:15:52.364 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:15.466475089Z", 2020-11-09T18:15:52.364 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.364 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.364 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.365 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.365 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.365 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.365 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_cxcDXnujWtafpwp1_Zf7_7E", 2020-11-09T18:15:52.365 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.365 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.365 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.366 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.366 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.366 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.366 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.366 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.366 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945595.470860_116.1", 2020-11-09T18:15:52.366 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.367 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-21", 2020-11-09T18:15:52.367 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:15.470860202Z", 2020-11-09T18:15:52.367 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.367 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.367 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.367 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.367 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.367 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.368 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_cxcDXnujWtafpwp1_Zf7_7E", 2020-11-09T18:15:52.368 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.368 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.368 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.368 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.368 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.368 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.369 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.369 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.369 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945595.574082_117.1", 2020-11-09T18:15:52.369 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.369 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-22:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.22", 2020-11-09T18:15:52.369 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:15.574082160Z", 2020-11-09T18:15:52.369 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.370 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.370 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.370 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.370 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.370 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.370 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_gu325NeoTQ-n6-N4RZO_Ufq", 2020-11-09T18:15:52.370 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.371 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.371 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.371 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.371 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.371 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.371 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.371 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.372 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945595.578263_118.1", 2020-11-09T18:15:52.372 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.372 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-22:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.22", 2020-11-09T18:15:52.372 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:15.578263069Z", 2020-11-09T18:15:52.372 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.372 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.372 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.372 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.373 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.373 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.373 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_gu325NeoTQ-n6-N4RZO_Ufq", 2020-11-09T18:15:52.373 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.373 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.373 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.373 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.374 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.374 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.374 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.374 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.374 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945595.580706_119.1", 2020-11-09T18:15:52.374 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.374 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-22", 2020-11-09T18:15:52.375 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:15.580706536Z", 2020-11-09T18:15:52.375 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.375 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.375 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.375 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.375 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.375 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.375 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_lHWYslSbF-coDhm_0XUYDqP", 2020-11-09T18:15:52.376 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.376 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.376 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.376 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.376 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.376 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.376 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.377 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.377 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945595.585376_120.1", 2020-11-09T18:15:52.377 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.377 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-22", 2020-11-09T18:15:52.377 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:15.585376092Z", 2020-11-09T18:15:52.377 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.377 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.377 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.378 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.378 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.378 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.378 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_lHWYslSbF-coDhm_0XUYDqP", 2020-11-09T18:15:52.378 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.378 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.378 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.379 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.379 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.379 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.379 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.379 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.379 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945595.651287_121.1", 2020-11-09T18:15:52.379 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.379 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-23:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.23", 2020-11-09T18:15:52.380 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:15.651287758Z", 2020-11-09T18:15:52.380 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.380 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.380 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.380 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.380 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.380 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.381 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_k8tGHuX9HMmJcr5VgC0V0M4", 2020-11-09T18:15:52.381 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.381 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.381 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.381 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.381 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.381 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.382 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.382 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.382 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945595.655554_122.1", 2020-11-09T18:15:52.382 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.382 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-23:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.23", 2020-11-09T18:15:52.382 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:15.655554732Z", 2020-11-09T18:15:52.382 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.382 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.383 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.383 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.383 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.383 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.383 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_k8tGHuX9HMmJcr5VgC0V0M4", 2020-11-09T18:15:52.383 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.384 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.384 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.384 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.384 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.384 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.384 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.384 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.385 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945595.657606_123.1", 2020-11-09T18:15:52.385 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.385 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-23", 2020-11-09T18:15:52.385 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:15.657606349Z", 2020-11-09T18:15:52.385 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.385 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.386 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.386 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.386 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.386 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.386 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_rDxg2fTzKxNRTnXPuogbkec", 2020-11-09T18:15:52.386 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.387 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.387 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.387 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.387 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.387 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.387 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.387 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.387 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945595.662266_124.1", 2020-11-09T18:15:52.388 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.388 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-23", 2020-11-09T18:15:52.388 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:15.662266951Z", 2020-11-09T18:15:52.388 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.388 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.388 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.388 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.389 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.389 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.389 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_rDxg2fTzKxNRTnXPuogbkec", 2020-11-09T18:15:52.389 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.389 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.389 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.389 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.389 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.390 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.390 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.390 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.390 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945595.694579_125.1", 2020-11-09T18:15:52.390 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.390 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-21:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.21", 2020-11-09T18:15:52.390 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:15.694579560Z", 2020-11-09T18:15:52.391 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.391 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.391 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_7weq2-_FEkT51PfhS1TopF2", 2020-11-09T18:15:52.391 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.391 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.391 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.391 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_7weq2-_FEkT51PfhS1TopF2", 2020-11-09T18:15:52.392 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.392 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.392 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.392 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.392 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.392 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.392 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.393 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.393 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945595.699216_126.1", 2020-11-09T18:15:52.393 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.393 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-21:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.21", 2020-11-09T18:15:52.393 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:15.699216478Z", 2020-11-09T18:15:52.393 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.393 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.394 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_7weq2-_FEkT51PfhS1TopF2", 2020-11-09T18:15:52.394 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.394 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.394 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.394 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_7weq2-_FEkT51PfhS1TopF2", 2020-11-09T18:15:52.394 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.394 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.395 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.395 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.395 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.395 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.395 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.395 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.395 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945595.704101_127.1", 2020-11-09T18:15:52.395 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.396 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-22:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.22", 2020-11-09T18:15:52.396 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:15.704101323Z", 2020-11-09T18:15:52.396 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.396 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.396 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_gu325NeoTQ-n6-N4RZO_Ufq", 2020-11-09T18:15:52.396 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.396 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.397 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.397 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_gu325NeoTQ-n6-N4RZO_Ufq", 2020-11-09T18:15:52.397 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.397 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.397 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.397 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.398 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.398 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.398 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.398 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.398 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945595.709327_128.1", 2020-11-09T18:15:52.399 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.399 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-22:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.22", 2020-11-09T18:15:52.399 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:15.709327694Z", 2020-11-09T18:15:52.399 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.399 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.399 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_gu325NeoTQ-n6-N4RZO_Ufq", 2020-11-09T18:15:52.400 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.400 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.400 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.400 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_gu325NeoTQ-n6-N4RZO_Ufq", 2020-11-09T18:15:52.400 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.400 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.400 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.401 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.401 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.401 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.401 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.401 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.401 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945595.722945_129.1", 2020-11-09T18:15:52.402 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.402 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-23:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.23", 2020-11-09T18:15:52.402 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:15.722945874Z", 2020-11-09T18:15:52.402 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.402 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.402 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_k8tGHuX9HMmJcr5VgC0V0M4", 2020-11-09T18:15:52.402 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.403 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.403 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.403 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_k8tGHuX9HMmJcr5VgC0V0M4", 2020-11-09T18:15:52.403 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.403 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.403 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.403 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.403 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.404 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.404 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.404 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.404 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945595.729371_130.1", 2020-11-09T18:15:52.404 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.404 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-23:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.23", 2020-11-09T18:15:52.404 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:15.729371675Z", 2020-11-09T18:15:52.405 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.405 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.405 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_k8tGHuX9HMmJcr5VgC0V0M4", 2020-11-09T18:15:52.405 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.405 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.405 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.405 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_k8tGHuX9HMmJcr5VgC0V0M4", 2020-11-09T18:15:52.405 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.406 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.406 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.406 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.406 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.406 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.406 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.406 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.407 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945597.242843_131.1", 2020-11-09T18:15:52.407 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.407 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-24:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.24", 2020-11-09T18:15:52.407 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:17.242843380Z", 2020-11-09T18:15:52.407 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.407 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.407 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.408 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.408 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.408 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.408 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_ZWaAaUyrIgJWu47a8QLknCB", 2020-11-09T18:15:52.408 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.408 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.408 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.408 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.409 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.409 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.409 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.409 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.409 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945597.247361_133.1", 2020-11-09T18:15:52.409 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.409 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-24:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.24", 2020-11-09T18:15:52.410 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:17.247361813Z", 2020-11-09T18:15:52.410 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.410 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.410 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.410 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.410 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.410 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.411 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_ZWaAaUyrIgJWu47a8QLknCB", 2020-11-09T18:15:52.411 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.411 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.411 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.411 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.411 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.411 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.412 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.412 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.412 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945597.249391_134.1", 2020-11-09T18:15:52.412 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.412 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-24", 2020-11-09T18:15:52.412 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:17.249391359Z", 2020-11-09T18:15:52.412 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.412 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.413 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.413 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.413 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.413 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.413 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_M6yzvodjSwrGlU0XXU23tmw", 2020-11-09T18:15:52.413 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.414 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.414 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.414 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.414 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.414 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.414 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.414 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.415 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945597.257906_135.1", 2020-11-09T18:15:52.415 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.415 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-24", 2020-11-09T18:15:52.415 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:17.257906751Z", 2020-11-09T18:15:52.415 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.415 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.415 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.416 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.416 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.416 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.416 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_M6yzvodjSwrGlU0XXU23tmw", 2020-11-09T18:15:52.416 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.416 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.417 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.417 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.417 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.417 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.417 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.417 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.417 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945598.021538_136.1", 2020-11-09T18:15:52.418 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.418 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-25:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.25", 2020-11-09T18:15:52.418 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:18.021538398Z", 2020-11-09T18:15:52.418 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.418 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.418 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.419 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.419 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.419 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.419 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dm5hw1Gq9w9JdDaVBV0h96_", 2020-11-09T18:15:52.419 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.419 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.420 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.420 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.420 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.420 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.420 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.420 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.421 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945598.026407_137.1", 2020-11-09T18:15:52.421 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.421 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-25:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.25", 2020-11-09T18:15:52.421 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:18.026407438Z", 2020-11-09T18:15:52.421 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.421 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.421 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.422 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.422 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.422 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.422 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dm5hw1Gq9w9JdDaVBV0h96_", 2020-11-09T18:15:52.422 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.422 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.422 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.423 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.423 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.423 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.423 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.423 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.423 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945598.028359_138.1", 2020-11-09T18:15:52.423 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.424 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-25", 2020-11-09T18:15:52.424 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:18.028359151Z", 2020-11-09T18:15:52.424 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.424 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.424 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.424 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.424 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.424 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.425 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_hJWxzyq2VXkrZe0lTr6UJ0X", 2020-11-09T18:15:52.425 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.425 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.425 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.425 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.425 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.425 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.426 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.426 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.426 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945598.032307_139.1", 2020-11-09T18:15:52.426 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.426 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-25", 2020-11-09T18:15:52.426 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:18.032307526Z", 2020-11-09T18:15:52.426 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.427 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.427 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.427 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.427 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.427 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.427 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_hJWxzyq2VXkrZe0lTr6UJ0X", 2020-11-09T18:15:52.427 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.427 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.428 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.428 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.428 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.428 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.428 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.428 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.428 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945598.347441_140.1", 2020-11-09T18:15:52.429 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.429 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-26:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.26", 2020-11-09T18:15:52.429 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:18.347441566Z", 2020-11-09T18:15:52.429 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.429 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.429 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.429 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.430 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.430 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.430 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_tUH5PREzBpkbjt0vmvKAgOc", 2020-11-09T18:15:52.430 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.430 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.430 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.430 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.430 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.431 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.431 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.431 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.431 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945598.351465_141.1", 2020-11-09T18:15:52.431 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.431 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-26:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.26", 2020-11-09T18:15:52.431 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:18.351465217Z", 2020-11-09T18:15:52.432 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.432 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.432 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.432 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.432 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.432 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.432 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_tUH5PREzBpkbjt0vmvKAgOc", 2020-11-09T18:15:52.433 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.433 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.433 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.433 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.433 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.433 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.433 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.434 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.434 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945598.353325_142.1", 2020-11-09T18:15:52.434 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.434 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-26", 2020-11-09T18:15:52.434 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:18.353325221Z", 2020-11-09T18:15:52.434 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.434 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.435 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.435 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.435 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.435 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.435 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_Z6tqFl2xPji_I9drTzIP2KZ", 2020-11-09T18:15:52.435 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.436 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.436 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.436 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.436 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.436 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.436 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.436 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.436 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945598.357392_143.1", 2020-11-09T18:15:52.437 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.437 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-26", 2020-11-09T18:15:52.437 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:18.357392493Z", 2020-11-09T18:15:52.437 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.437 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.437 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.437 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.438 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.438 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.438 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_Z6tqFl2xPji_I9drTzIP2KZ", 2020-11-09T18:15:52.438 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.438 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.438 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.438 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.439 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.439 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.439 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.439 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.439 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945598.450979_144.1", 2020-11-09T18:15:52.439 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.439 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-24:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.24", 2020-11-09T18:15:52.440 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:18.450979415Z", 2020-11-09T18:15:52.440 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.440 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.440 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_ZWaAaUyrIgJWu47a8QLknCB", 2020-11-09T18:15:52.440 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.440 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.441 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.441 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_ZWaAaUyrIgJWu47a8QLknCB", 2020-11-09T18:15:52.441 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.441 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.441 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.441 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.441 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.442 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.442 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.442 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.442 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945598.461497_145.1", 2020-11-09T18:15:52.442 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.442 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-24:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.24", 2020-11-09T18:15:52.443 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:18.461497100Z", 2020-11-09T18:15:52.443 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.443 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.443 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_ZWaAaUyrIgJWu47a8QLknCB", 2020-11-09T18:15:52.443 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.443 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.444 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.444 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_ZWaAaUyrIgJWu47a8QLknCB", 2020-11-09T18:15:52.444 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.444 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.444 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.444 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.444 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.445 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.445 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.445 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.445 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945598.470055_146.1", 2020-11-09T18:15:52.445 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.445 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-24:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.24", 2020-11-09T18:15:52.446 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:18.470055992Z", 2020-11-09T18:15:52.446 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.446 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.446 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_ZWaAaUyrIgJWu47a8QLknCB", 2020-11-09T18:15:52.446 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.446 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.446 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.447 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_ZWaAaUyrIgJWu47a8QLknCB", 2020-11-09T18:15:52.447 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 3 2020-11-09T18:15:52.447 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.447 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.447 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.447 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.447 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.448 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.448 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.448 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945598.478206_147.1", 2020-11-09T18:15:52.448 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.448 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-24:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.24", 2020-11-09T18:15:52.448 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:18.478206910Z", 2020-11-09T18:15:52.448 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.449 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.449 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_ZWaAaUyrIgJWu47a8QLknCB", 2020-11-09T18:15:52.449 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.449 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.449 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.449 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_ZWaAaUyrIgJWu47a8QLknCB", 2020-11-09T18:15:52.449 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 3 2020-11-09T18:15:52.449 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.450 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.450 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.450 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.450 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.450 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.450 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.450 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945598.482582_148.1", 2020-11-09T18:15:52.451 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.451 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-25:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.25", 2020-11-09T18:15:52.451 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:18.482582905Z", 2020-11-09T18:15:52.451 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.451 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.451 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dm5hw1Gq9w9JdDaVBV0h96_", 2020-11-09T18:15:52.451 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.452 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.452 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.452 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dm5hw1Gq9w9JdDaVBV0h96_", 2020-11-09T18:15:52.452 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.452 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.452 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.452 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.453 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.453 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.453 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.453 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.453 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945598.488285_149.1", 2020-11-09T18:15:52.453 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.453 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-25:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.25", 2020-11-09T18:15:52.454 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:18.488285331Z", 2020-11-09T18:15:52.454 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.454 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.454 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dm5hw1Gq9w9JdDaVBV0h96_", 2020-11-09T18:15:52.454 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.454 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.454 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.455 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dm5hw1Gq9w9JdDaVBV0h96_", 2020-11-09T18:15:52.455 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.455 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.455 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.455 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.455 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.455 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.456 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.456 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.456 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945598.495708_150.1", 2020-11-09T18:15:52.456 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.456 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-25:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.25", 2020-11-09T18:15:52.456 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:18.495708983Z", 2020-11-09T18:15:52.456 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.457 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.457 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dm5hw1Gq9w9JdDaVBV0h96_", 2020-11-09T18:15:52.457 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.457 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.457 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.457 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dm5hw1Gq9w9JdDaVBV0h96_", 2020-11-09T18:15:52.457 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 3 2020-11-09T18:15:52.458 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.458 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.458 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.458 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.458 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.458 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.458 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.459 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945598.502185_151.1", 2020-11-09T18:15:52.459 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.459 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-25:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.25", 2020-11-09T18:15:52.459 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:18.502185684Z", 2020-11-09T18:15:52.459 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.459 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.460 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dm5hw1Gq9w9JdDaVBV0h96_", 2020-11-09T18:15:52.460 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.460 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.460 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.460 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dm5hw1Gq9w9JdDaVBV0h96_", 2020-11-09T18:15:52.460 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 3 2020-11-09T18:15:52.460 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.461 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.461 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.461 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.461 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.461 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.461 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.461 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945598.514968_152.1", 2020-11-09T18:15:52.462 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.462 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-26:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.26", 2020-11-09T18:15:52.462 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:18.514968069Z", 2020-11-09T18:15:52.462 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.462 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.462 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_tUH5PREzBpkbjt0vmvKAgOc", 2020-11-09T18:15:52.463 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.463 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.463 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.463 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_tUH5PREzBpkbjt0vmvKAgOc", 2020-11-09T18:15:52.463 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.463 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.463 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.464 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.464 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.464 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.464 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.464 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.464 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945598.528310_153.1", 2020-11-09T18:15:52.465 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.465 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-26:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.26", 2020-11-09T18:15:52.465 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:18.528310207Z", 2020-11-09T18:15:52.465 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.465 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.465 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_tUH5PREzBpkbjt0vmvKAgOc", 2020-11-09T18:15:52.465 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.466 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.466 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.466 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_tUH5PREzBpkbjt0vmvKAgOc", 2020-11-09T18:15:52.466 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.466 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.466 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.466 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.467 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.467 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.467 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.467 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.467 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945598.543001_154.1", 2020-11-09T18:15:52.467 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.467 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-26:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.26", 2020-11-09T18:15:52.468 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:18.543001147Z", 2020-11-09T18:15:52.468 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.468 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.468 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_tUH5PREzBpkbjt0vmvKAgOc", 2020-11-09T18:15:52.468 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.468 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.469 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.469 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_tUH5PREzBpkbjt0vmvKAgOc", 2020-11-09T18:15:52.469 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 3 2020-11-09T18:15:52.469 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.469 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.469 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.470 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.470 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.470 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.470 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.470 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945598.548940_155.1", 2020-11-09T18:15:52.470 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.471 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-26:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.26", 2020-11-09T18:15:52.471 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:18.548940287Z", 2020-11-09T18:15:52.471 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.471 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.471 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_tUH5PREzBpkbjt0vmvKAgOc", 2020-11-09T18:15:52.471 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.471 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.472 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.472 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_tUH5PREzBpkbjt0vmvKAgOc", 2020-11-09T18:15:52.472 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 3 2020-11-09T18:15:52.472 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.472 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.472 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.473 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.473 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.473 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.473 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.473 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945600.502084_156.1", 2020-11-09T18:15:52.473 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.473 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-27:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.27", 2020-11-09T18:15:52.474 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:20.502084837Z", 2020-11-09T18:15:52.474 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.474 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.474 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.474 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.474 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.475 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.475 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_ffjKv4uq3V4IpykJfSlfwhB", 2020-11-09T18:15:52.475 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.475 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.475 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.475 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.476 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.476 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.476 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.476 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.476 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945600.507516_158.1", 2020-11-09T18:15:52.476 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.476 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-27:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.27", 2020-11-09T18:15:52.477 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:20.507516916Z", 2020-11-09T18:15:52.477 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.477 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.477 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.477 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.477 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.478 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.478 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_ffjKv4uq3V4IpykJfSlfwhB", 2020-11-09T18:15:52.478 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.478 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.478 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.478 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.479 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.479 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.479 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.479 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.479 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945600.509661_159.1", 2020-11-09T18:15:52.479 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.480 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-27", 2020-11-09T18:15:52.480 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:20.509661301Z", 2020-11-09T18:15:52.480 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.480 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.480 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.480 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.481 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.481 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.481 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_UJQ25D20xb-Xxms2LevItvh", 2020-11-09T18:15:52.481 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.481 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.481 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.481 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.482 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.482 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.482 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.482 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.482 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945600.513944_160.1", 2020-11-09T18:15:52.482 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.483 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-27", 2020-11-09T18:15:52.483 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:20.513944844Z", 2020-11-09T18:15:52.483 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.483 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.483 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.483 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.484 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.484 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.484 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_UJQ25D20xb-Xxms2LevItvh", 2020-11-09T18:15:52.484 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.484 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.484 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.485 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.485 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.485 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.485 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.485 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.485 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945601.434311_161.1", 2020-11-09T18:15:52.485 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.486 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-28:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.28", 2020-11-09T18:15:52.486 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:21.434311607Z", 2020-11-09T18:15:52.486 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.486 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.486 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.487 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.487 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.487 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.487 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_jFhIAKXmQdsa5R34lXgk3pv", 2020-11-09T18:15:52.487 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.487 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.487 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.488 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.488 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.488 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.488 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.488 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.488 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945601.438688_162.1", 2020-11-09T18:15:52.488 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.489 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-28:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.28", 2020-11-09T18:15:52.489 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:21.438688009Z", 2020-11-09T18:15:52.489 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.489 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.489 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.489 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.489 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.490 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.490 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_jFhIAKXmQdsa5R34lXgk3pv", 2020-11-09T18:15:52.490 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.490 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.490 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.490 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.490 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.491 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.491 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.491 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.491 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945601.440791_163.1", 2020-11-09T18:15:52.491 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.491 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-28", 2020-11-09T18:15:52.491 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:21.440791127Z", 2020-11-09T18:15:52.491 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.492 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.492 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.492 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.492 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.492 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.492 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_4xBDSjdegdCsfVupUBmzzRX", 2020-11-09T18:15:52.493 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.493 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.493 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.493 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.493 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.493 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.494 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.494 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.494 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945601.445125_164.1", 2020-11-09T18:15:52.494 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.494 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-28", 2020-11-09T18:15:52.494 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:21.445125336Z", 2020-11-09T18:15:52.495 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.495 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.495 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.495 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.495 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.495 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.496 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_4xBDSjdegdCsfVupUBmzzRX", 2020-11-09T18:15:52.496 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.496 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.496 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.496 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.496 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.496 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.497 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.497 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.497 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945603.023496_165.1", 2020-11-09T18:15:52.497 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.497 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-29:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.29", 2020-11-09T18:15:52.497 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:23.023496753Z", 2020-11-09T18:15:52.498 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.498 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.498 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.498 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.498 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.498 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.499 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_P83ahmV5nZ3tNcs0_lSHbDU", 2020-11-09T18:15:52.499 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.499 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.499 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.499 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.499 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.499 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.500 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.500 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.500 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945603.027781_166.1", 2020-11-09T18:15:52.500 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.500 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-29:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.29", 2020-11-09T18:15:52.500 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:23.027781495Z", 2020-11-09T18:15:52.500 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.501 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.501 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.501 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.501 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.501 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.501 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_P83ahmV5nZ3tNcs0_lSHbDU", 2020-11-09T18:15:52.502 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.502 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.502 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.502 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.502 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.502 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.503 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.503 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.503 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945603.030309_167.1", 2020-11-09T18:15:52.503 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.503 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-29", 2020-11-09T18:15:52.503 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:23.030309393Z", 2020-11-09T18:15:52.504 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.504 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.504 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.504 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.504 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.504 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.505 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_Hhh1A6pFRNzrIunHlFDZwos", 2020-11-09T18:15:52.505 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.505 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.505 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.505 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.505 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.505 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.506 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.506 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.506 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945603.034891_168.1", 2020-11-09T18:15:52.506 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.506 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-29", 2020-11-09T18:15:52.506 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:23.034891461Z", 2020-11-09T18:15:52.506 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.507 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.507 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.507 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.507 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.507 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.507 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_Hhh1A6pFRNzrIunHlFDZwos", 2020-11-09T18:15:52.507 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.507 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.508 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.508 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.508 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.508 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.508 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.508 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.508 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945603.213259_169.1", 2020-11-09T18:15:52.509 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.509 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-27:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.27", 2020-11-09T18:15:52.509 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:23.213259709Z", 2020-11-09T18:15:52.509 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.509 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.509 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_ffjKv4uq3V4IpykJfSlfwhB", 2020-11-09T18:15:52.509 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.510 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.510 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.510 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_ffjKv4uq3V4IpykJfSlfwhB", 2020-11-09T18:15:52.510 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.510 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.510 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.510 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.511 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.511 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.511 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.511 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.511 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945603.223295_170.1", 2020-11-09T18:15:52.511 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.511 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-27:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.27", 2020-11-09T18:15:52.512 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:23.223295591Z", 2020-11-09T18:15:52.517 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.518 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.518 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_ffjKv4uq3V4IpykJfSlfwhB", 2020-11-09T18:15:52.518 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.518 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.518 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.518 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_ffjKv4uq3V4IpykJfSlfwhB", 2020-11-09T18:15:52.518 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.519 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.519 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.519 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.519 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.519 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.519 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.519 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.520 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945604.018547_171.1", 2020-11-09T18:15:52.520 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.520 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-28:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.28", 2020-11-09T18:15:52.520 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:24.018547219Z", 2020-11-09T18:15:52.520 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.520 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.521 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_jFhIAKXmQdsa5R34lXgk3pv", 2020-11-09T18:15:52.521 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.521 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.521 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.521 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_jFhIAKXmQdsa5R34lXgk3pv", 2020-11-09T18:15:52.521 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.521 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.522 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.522 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.522 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.522 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.522 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.522 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.523 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945604.029428_172.1", 2020-11-09T18:15:52.523 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.523 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-28:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.28", 2020-11-09T18:15:52.523 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:24.029428921Z", 2020-11-09T18:15:52.523 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.523 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.523 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_jFhIAKXmQdsa5R34lXgk3pv", 2020-11-09T18:15:52.524 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.524 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.524 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.524 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_jFhIAKXmQdsa5R34lXgk3pv", 2020-11-09T18:15:52.524 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.524 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.525 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.525 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.525 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.525 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.525 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.525 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.525 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945604.720659_173.1", 2020-11-09T18:15:52.526 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.526 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-29:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.29", 2020-11-09T18:15:52.526 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:24.720659097Z", 2020-11-09T18:15:52.526 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.526 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.526 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_P83ahmV5nZ3tNcs0_lSHbDU", 2020-11-09T18:15:52.527 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.527 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.527 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.527 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_P83ahmV5nZ3tNcs0_lSHbDU", 2020-11-09T18:15:52.527 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.527 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.528 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.528 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.528 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.528 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.528 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.528 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.529 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945604.732717_174.1", 2020-11-09T18:15:52.529 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.529 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-29:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.29", 2020-11-09T18:15:52.529 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:24.732717989Z", 2020-11-09T18:15:52.529 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.529 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.529 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_P83ahmV5nZ3tNcs0_lSHbDU", 2020-11-09T18:15:52.530 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.530 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.530 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.530 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_P83ahmV5nZ3tNcs0_lSHbDU", 2020-11-09T18:15:52.530 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.530 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.530 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.531 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.531 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.531 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.531 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.531 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.531 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945605.406938_175.1", 2020-11-09T18:15:52.532 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.532 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-30:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.30", 2020-11-09T18:15:52.532 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:25.406938541Z", 2020-11-09T18:15:52.532 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.532 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.532 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.533 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.533 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.533 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.533 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_2k6PaixusxbzBO-jYw6D0Md", 2020-11-09T18:15:52.533 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.533 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.533 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.534 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.534 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.534 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.534 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.534 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.534 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945605.412027_176.1", 2020-11-09T18:15:52.535 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.535 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-30:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.30", 2020-11-09T18:15:52.535 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:25.412027440Z", 2020-11-09T18:15:52.535 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.535 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.535 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.535 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.536 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.536 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.536 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_2k6PaixusxbzBO-jYw6D0Md", 2020-11-09T18:15:52.536 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.536 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.536 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.537 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.537 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.537 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.537 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.537 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.537 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945605.413981_177.1", 2020-11-09T18:15:52.538 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.538 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-30", 2020-11-09T18:15:52.538 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:25.413981551Z", 2020-11-09T18:15:52.538 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.538 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.538 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.538 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.539 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.539 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.539 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_OVN11Cxzp_HRsHiBhSltQ2E", 2020-11-09T18:15:52.539 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.539 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.539 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.540 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.540 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.540 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.540 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.540 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.540 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945605.419171_178.1", 2020-11-09T18:15:52.541 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.541 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-30", 2020-11-09T18:15:52.541 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:25.419171989Z", 2020-11-09T18:15:52.541 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.541 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.541 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.541 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.542 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.542 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.542 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_OVN11Cxzp_HRsHiBhSltQ2E", 2020-11-09T18:15:52.542 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.542 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.542 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.543 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.543 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.543 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.543 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.543 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.543 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945605.813895_179.1", 2020-11-09T18:15:52.544 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.544 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-31:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.31", 2020-11-09T18:15:52.544 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:25.813895626Z", 2020-11-09T18:15:52.544 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.544 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.544 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.544 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.545 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.545 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.545 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dboGSnUFvBxtlhGOt6TYcuV", 2020-11-09T18:15:52.545 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.545 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.545 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.546 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.546 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.546 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.546 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.546 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.546 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945605.819203_180.1", 2020-11-09T18:15:52.547 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.547 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-31:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.31", 2020-11-09T18:15:52.547 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:25.819203440Z", 2020-11-09T18:15:52.547 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.547 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.547 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.547 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.548 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.548 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.548 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dboGSnUFvBxtlhGOt6TYcuV", 2020-11-09T18:15:52.548 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.548 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.548 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.548 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.549 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.549 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.549 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.549 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.549 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945605.821244_181.1", 2020-11-09T18:15:52.549 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.549 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-31", 2020-11-09T18:15:52.549 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:25.821244081Z", 2020-11-09T18:15:52.550 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.550 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.550 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.550 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.550 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.550 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.550 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_D_zBqUuf_PUMl6KPscj9tcq", 2020-11-09T18:15:52.551 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.551 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.551 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.551 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.551 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.551 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.551 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.551 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.552 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945605.825854_182.1", 2020-11-09T18:15:52.552 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.552 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-31", 2020-11-09T18:15:52.552 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:25.825854764Z", 2020-11-09T18:15:52.552 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.552 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.552 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.553 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.553 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.553 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.553 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_D_zBqUuf_PUMl6KPscj9tcq", 2020-11-09T18:15:52.553 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.553 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.553 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.553 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.554 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.554 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.554 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.554 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.554 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.314889_183.1", 2020-11-09T18:15:52.554 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.554 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-32:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.32", 2020-11-09T18:15:52.555 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.314889240Z", 2020-11-09T18:15:52.555 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.555 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.555 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.555 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.555 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.555 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.555 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_h7iYndUz6ZZeJDRnCAJhq6b", 2020-11-09T18:15:52.556 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.556 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.556 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.556 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.556 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.556 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.556 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.557 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.557 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.319791_184.1", 2020-11-09T18:15:52.557 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.557 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-32:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.32", 2020-11-09T18:15:52.557 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.319791421Z", 2020-11-09T18:15:52.557 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.557 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.557 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.558 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.558 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.558 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.558 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_h7iYndUz6ZZeJDRnCAJhq6b", 2020-11-09T18:15:52.558 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.558 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.558 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.559 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.559 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.559 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.559 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.559 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.559 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.321799_185.1", 2020-11-09T18:15:52.559 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.560 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-32", 2020-11-09T18:15:52.560 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.321799710Z", 2020-11-09T18:15:52.560 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.560 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.560 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.560 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.560 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.560 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.561 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_kfgLDq_uLGT_Ii4AzNlTUJd", 2020-11-09T18:15:52.561 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.561 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.561 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.561 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.561 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.561 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.562 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.562 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.562 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.326129_186.1", 2020-11-09T18:15:52.562 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.562 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-32", 2020-11-09T18:15:52.562 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.326129812Z", 2020-11-09T18:15:52.562 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.563 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.563 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.563 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.563 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.563 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.563 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_kfgLDq_uLGT_Ii4AzNlTUJd", 2020-11-09T18:15:52.563 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.563 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.564 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.564 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.564 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.564 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.564 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.564 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.564 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.351846_187.1", 2020-11-09T18:15:52.565 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.565 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-30:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.30", 2020-11-09T18:15:52.565 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.351846735Z", 2020-11-09T18:15:52.565 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.565 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.565 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_2k6PaixusxbzBO-jYw6D0Md", 2020-11-09T18:15:52.565 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.565 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.566 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.566 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_2k6PaixusxbzBO-jYw6D0Md", 2020-11-09T18:15:52.566 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.566 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.566 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.566 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.566 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.567 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.567 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.567 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.567 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.358962_188.1", 2020-11-09T18:15:52.567 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.567 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-30:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.30", 2020-11-09T18:15:52.567 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.358962403Z", 2020-11-09T18:15:52.568 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.568 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.568 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_2k6PaixusxbzBO-jYw6D0Md", 2020-11-09T18:15:52.568 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.568 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.568 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.568 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_2k6PaixusxbzBO-jYw6D0Md", 2020-11-09T18:15:52.569 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.569 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.569 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.569 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.569 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.569 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.569 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.570 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.570 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.413199_189.1", 2020-11-09T18:15:52.570 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.570 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-31:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.31", 2020-11-09T18:15:52.570 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.413199050Z", 2020-11-09T18:15:52.570 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.570 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.571 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dboGSnUFvBxtlhGOt6TYcuV", 2020-11-09T18:15:52.571 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.571 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.571 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.571 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dboGSnUFvBxtlhGOt6TYcuV", 2020-11-09T18:15:52.571 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.571 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.572 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.572 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.572 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.572 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.572 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.572 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.573 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.423298_190.1", 2020-11-09T18:15:52.573 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.573 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-31:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.31", 2020-11-09T18:15:52.573 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.423298203Z", 2020-11-09T18:15:52.573 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.573 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.573 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dboGSnUFvBxtlhGOt6TYcuV", 2020-11-09T18:15:52.574 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.574 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.574 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.574 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dboGSnUFvBxtlhGOt6TYcuV", 2020-11-09T18:15:52.574 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.574 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.575 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.575 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.575 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.575 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.575 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.575 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.575 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.486367_191.1", 2020-11-09T18:15:52.576 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.576 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-32:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.32", 2020-11-09T18:15:52.576 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.486367158Z", 2020-11-09T18:15:52.576 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.576 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.576 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_h7iYndUz6ZZeJDRnCAJhq6b", 2020-11-09T18:15:52.577 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.577 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.577 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.577 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_h7iYndUz6ZZeJDRnCAJhq6b", 2020-11-09T18:15:52.577 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.577 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.577 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.578 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.578 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.578 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.578 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.578 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.578 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.497483_192.1", 2020-11-09T18:15:52.579 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.579 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-32:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.32", 2020-11-09T18:15:52.579 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.497483092Z", 2020-11-09T18:15:52.579 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.579 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.579 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_h7iYndUz6ZZeJDRnCAJhq6b", 2020-11-09T18:15:52.579 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.579 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.580 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.580 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_h7iYndUz6ZZeJDRnCAJhq6b", 2020-11-09T18:15:52.580 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.580 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.580 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.580 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.580 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.581 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.581 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.581 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.581 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.567936_193.1", 2020-11-09T18:15:52.581 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.581 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-33:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.33", 2020-11-09T18:15:52.581 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.567936942Z", 2020-11-09T18:15:52.582 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.582 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.582 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.582 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.582 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.582 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.582 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dV7Zwb-EMDhOHgweLJlLD79", 2020-11-09T18:15:52.583 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.583 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.583 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.583 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.583 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.583 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.583 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.584 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.584 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.573179_194.1", 2020-11-09T18:15:52.584 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.584 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-33:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.33", 2020-11-09T18:15:52.584 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.573179777Z", 2020-11-09T18:15:52.584 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.584 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.584 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.585 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.585 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.585 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.585 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dV7Zwb-EMDhOHgweLJlLD79", 2020-11-09T18:15:52.585 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.585 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.585 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.586 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.586 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.586 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.586 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.586 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.586 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.575137_195.1", 2020-11-09T18:15:52.587 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.587 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-33", 2020-11-09T18:15:52.587 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.575137143Z", 2020-11-09T18:15:52.587 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.587 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.587 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.587 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.588 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.588 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.588 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "__9pytw2itoqGRYgrLwyaE0v", 2020-11-09T18:15:52.588 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.588 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.588 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.588 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.589 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.589 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.589 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.589 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.589 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.578921_196.1", 2020-11-09T18:15:52.589 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.590 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-33", 2020-11-09T18:15:52.590 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.578921064Z", 2020-11-09T18:15:52.590 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.590 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.590 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.590 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.590 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.591 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.591 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "__9pytw2itoqGRYgrLwyaE0v", 2020-11-09T18:15:52.591 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.591 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.591 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.591 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.591 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.591 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.592 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.592 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.592 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.603950_198.1", 2020-11-09T18:15:52.592 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.592 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-34:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.34", 2020-11-09T18:15:52.592 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.603950696Z", 2020-11-09T18:15:52.592 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.593 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.593 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.593 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.593 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.593 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.593 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_KazzTgKF9mHRl9t-mjqUZIF", 2020-11-09T18:15:52.593 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.593 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.594 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.594 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.594 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.594 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.594 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.594 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.594 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.608737_199.1", 2020-11-09T18:15:52.595 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.595 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-34:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.34", 2020-11-09T18:15:52.595 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.608737440Z", 2020-11-09T18:15:52.595 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.595 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.595 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.595 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.596 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.596 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.596 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_KazzTgKF9mHRl9t-mjqUZIF", 2020-11-09T18:15:52.596 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.596 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.596 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.596 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.596 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.597 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.597 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.597 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.597 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.610541_200.1", 2020-11-09T18:15:52.597 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.597 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-34", 2020-11-09T18:15:52.597 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.610541395Z", 2020-11-09T18:15:52.598 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.598 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.598 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.598 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.598 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.598 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.598 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_QkHzIdBeCUUuKSc9kaJ3NT5", 2020-11-09T18:15:52.598 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.599 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.599 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.599 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.599 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.599 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.599 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.599 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.600 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.614352_201.1", 2020-11-09T18:15:52.600 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.600 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-34", 2020-11-09T18:15:52.600 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.614352129Z", 2020-11-09T18:15:52.600 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.600 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.600 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.601 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.601 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.601 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.601 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_QkHzIdBeCUUuKSc9kaJ3NT5", 2020-11-09T18:15:52.601 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.601 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.601 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.601 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.602 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.602 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.602 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.602 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.602 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.658773_202.1", 2020-11-09T18:15:52.602 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.602 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-35:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.35", 2020-11-09T18:15:52.603 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.658773780Z", 2020-11-09T18:15:52.603 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.603 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.603 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.603 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.603 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.603 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.603 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dAiojqzewqe1JboY85aFSyJ", 2020-11-09T18:15:52.604 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.604 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.604 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.604 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.604 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.604 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.604 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.605 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.605 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.663192_203.1", 2020-11-09T18:15:52.605 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.605 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-35:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.35", 2020-11-09T18:15:52.605 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.663192679Z", 2020-11-09T18:15:52.605 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.605 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.606 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.606 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.606 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.606 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.606 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dAiojqzewqe1JboY85aFSyJ", 2020-11-09T18:15:52.606 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.606 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.606 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.607 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.607 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.607 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.607 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.607 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.607 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.665284_204.1", 2020-11-09T18:15:52.607 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.608 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-35", 2020-11-09T18:15:52.608 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.665284756Z", 2020-11-09T18:15:52.608 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.608 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.608 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.608 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.608 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.608 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.609 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_YIG4M8hwsEikvpJIsO0cCiX", 2020-11-09T18:15:52.609 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.609 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.609 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.609 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.609 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.609 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.610 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.610 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.610 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.671390_205.1", 2020-11-09T18:15:52.610 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.610 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-35", 2020-11-09T18:15:52.610 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.671390803Z", 2020-11-09T18:15:52.610 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.610 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.611 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.611 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.611 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.611 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.611 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_YIG4M8hwsEikvpJIsO0cCiX", 2020-11-09T18:15:52.611 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.611 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.612 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.612 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.612 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.612 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.612 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.612 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.612 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.708610_206.1", 2020-11-09T18:15:52.612 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.613 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-33:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.33", 2020-11-09T18:15:52.613 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.708610357Z", 2020-11-09T18:15:52.613 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.613 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.613 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dV7Zwb-EMDhOHgweLJlLD79", 2020-11-09T18:15:52.613 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.613 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.614 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.614 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dV7Zwb-EMDhOHgweLJlLD79", 2020-11-09T18:15:52.614 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.614 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.614 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.614 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.614 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.615 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.615 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.615 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.615 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945606.713328_207.1", 2020-11-09T18:15:52.615 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.615 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-33:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.33", 2020-11-09T18:15:52.615 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:26.713328296Z", 2020-11-09T18:15:52.615 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.616 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.616 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dV7Zwb-EMDhOHgweLJlLD79", 2020-11-09T18:15:52.616 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.616 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.616 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.616 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dV7Zwb-EMDhOHgweLJlLD79", 2020-11-09T18:15:52.616 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.617 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.617 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.617 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.617 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.617 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.617 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.617 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.617 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945607.760169_210.1", 2020-11-09T18:15:52.618 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.618 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-34:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.34", 2020-11-09T18:15:52.618 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:27.760169945Z", 2020-11-09T18:15:52.618 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.618 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.618 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_KazzTgKF9mHRl9t-mjqUZIF", 2020-11-09T18:15:52.618 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.619 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.619 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.619 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_KazzTgKF9mHRl9t-mjqUZIF", 2020-11-09T18:15:52.619 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.619 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.619 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.619 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.619 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.620 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.620 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.620 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.620 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945607.771847_211.1", 2020-11-09T18:15:52.620 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.620 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-34:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.34", 2020-11-09T18:15:52.620 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:27.771847568Z", 2020-11-09T18:15:52.621 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.621 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.621 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_KazzTgKF9mHRl9t-mjqUZIF", 2020-11-09T18:15:52.621 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.621 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.621 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.621 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_KazzTgKF9mHRl9t-mjqUZIF", 2020-11-09T18:15:52.622 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.622 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.622 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.622 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.622 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.622 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.622 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.622 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.623 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945609.277502_212.1", 2020-11-09T18:15:52.623 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.623 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-35:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.35", 2020-11-09T18:15:52.623 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:29.277502241Z", 2020-11-09T18:15:52.623 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.623 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.623 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dAiojqzewqe1JboY85aFSyJ", 2020-11-09T18:15:52.624 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.624 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.624 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.624 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dAiojqzewqe1JboY85aFSyJ", 2020-11-09T18:15:52.624 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.624 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.624 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.625 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.625 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.625 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.625 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.625 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.625 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945609.288552_213.1", 2020-11-09T18:15:52.625 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.625 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-35:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.35", 2020-11-09T18:15:52.626 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:29.288552016Z", 2020-11-09T18:15:52.626 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.626 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.626 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dAiojqzewqe1JboY85aFSyJ", 2020-11-09T18:15:52.626 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.626 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.626 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.627 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dAiojqzewqe1JboY85aFSyJ", 2020-11-09T18:15:52.627 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.627 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.627 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.627 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.627 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.627 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.627 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.628 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.628 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945609.660249_214.1", 2020-11-09T18:15:52.628 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.628 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-36:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.36", 2020-11-09T18:15:52.628 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:29.660249505Z", 2020-11-09T18:15:52.628 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.628 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.629 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.629 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.629 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.629 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.629 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_ozJuqjR3GVh8aGnB2Xh4hR5", 2020-11-09T18:15:52.629 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.629 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.629 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.630 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.630 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.630 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.630 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.630 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.630 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945609.665942_215.1", 2020-11-09T18:15:52.630 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.631 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-36:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.36", 2020-11-09T18:15:52.631 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:29.665942367Z", 2020-11-09T18:15:52.631 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.631 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.631 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.631 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.631 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.631 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.632 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_ozJuqjR3GVh8aGnB2Xh4hR5", 2020-11-09T18:15:52.632 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.632 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.632 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.632 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.632 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.632 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.633 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.633 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.633 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945609.669229_216.1", 2020-11-09T18:15:52.633 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.633 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-36", 2020-11-09T18:15:52.633 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:29.669229416Z", 2020-11-09T18:15:52.633 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.633 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.634 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.634 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.634 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.634 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.634 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_DZ1KbRgq_ERertkZGzu4RA_", 2020-11-09T18:15:52.634 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.634 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.635 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.635 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.635 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.635 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.635 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.635 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.635 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945609.675301_217.1", 2020-11-09T18:15:52.636 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.636 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-36", 2020-11-09T18:15:52.636 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:29.675301279Z", 2020-11-09T18:15:52.636 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.636 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.636 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.636 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.636 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.637 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.637 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_DZ1KbRgq_ERertkZGzu4RA_", 2020-11-09T18:15:52.637 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.637 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.637 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.637 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.637 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.638 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.638 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.638 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.638 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945610.079342_218.1", 2020-11-09T18:15:52.638 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.638 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-37:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.37", 2020-11-09T18:15:52.638 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:30.079342479Z", 2020-11-09T18:15:52.639 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.639 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.639 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.639 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.639 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.639 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.639 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_fx1DINtpIECPzOGaSfs3ucL", 2020-11-09T18:15:52.639 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.640 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.640 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.640 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.640 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.640 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.640 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.640 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.641 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945610.083364_219.1", 2020-11-09T18:15:52.641 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.641 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-37:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.37", 2020-11-09T18:15:52.641 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:30.083364883Z", 2020-11-09T18:15:52.641 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.641 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.641 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.642 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.642 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.642 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.642 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_fx1DINtpIECPzOGaSfs3ucL", 2020-11-09T18:15:52.642 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.642 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.642 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.642 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.643 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.643 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.643 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.643 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.643 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945610.085275_220.1", 2020-11-09T18:15:52.643 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.643 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-37", 2020-11-09T18:15:52.644 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:30.085275759Z", 2020-11-09T18:15:52.644 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.644 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.644 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.644 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.644 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.644 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.644 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_J2nfKVSwwp-9vo2WCGxBAhy", 2020-11-09T18:15:52.645 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.645 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.645 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.645 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.645 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.645 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.646 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.646 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.646 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945610.091583_221.1", 2020-11-09T18:15:52.646 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.646 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-37", 2020-11-09T18:15:52.646 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:30.091583370Z", 2020-11-09T18:15:52.646 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.646 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.647 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.647 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.647 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.647 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.647 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_J2nfKVSwwp-9vo2WCGxBAhy", 2020-11-09T18:15:52.647 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.647 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.648 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.648 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.648 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.648 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.648 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.648 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.648 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945610.137385_222.1", 2020-11-09T18:15:52.648 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.649 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-38:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.38", 2020-11-09T18:15:52.649 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:30.137385505Z", 2020-11-09T18:15:52.649 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.649 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.649 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.649 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.649 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.650 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.650 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_MKdJSqznasfgUfD8pSqNyMN", 2020-11-09T18:15:52.650 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.650 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.650 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.650 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.650 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.650 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.651 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.651 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.651 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945610.141970_223.1", 2020-11-09T18:15:52.651 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.651 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-38:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.38", 2020-11-09T18:15:52.651 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:30.141970597Z", 2020-11-09T18:15:52.651 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.652 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.652 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.652 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.652 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.652 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.652 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_MKdJSqznasfgUfD8pSqNyMN", 2020-11-09T18:15:52.652 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.652 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.653 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.653 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.653 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.653 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.653 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.653 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.653 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945610.144161_224.1", 2020-11-09T18:15:52.654 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.654 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-38", 2020-11-09T18:15:52.654 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:30.144161714Z", 2020-11-09T18:15:52.654 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.654 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.654 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.654 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.655 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.655 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.655 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_Lsx6lFhkZQ0sbnB7SgpHP7X", 2020-11-09T18:15:52.655 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.655 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.655 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.655 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.655 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.656 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.656 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.656 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.656 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945610.148613_225.1", 2020-11-09T18:15:52.656 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.656 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-38", 2020-11-09T18:15:52.656 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:30.148613440Z", 2020-11-09T18:15:52.657 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.657 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.657 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.657 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.657 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.657 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.657 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_Lsx6lFhkZQ0sbnB7SgpHP7X", 2020-11-09T18:15:52.657 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.658 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.658 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.658 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.658 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.658 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.658 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.658 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.659 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945611.763559_226.1", 2020-11-09T18:15:52.659 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.659 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-39:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.39", 2020-11-09T18:15:52.659 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:31.763559546Z", 2020-11-09T18:15:52.659 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.659 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.659 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.659 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.660 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.660 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.660 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_DoCr53MUjfmeA1PTWsLAoZg", 2020-11-09T18:15:52.660 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.660 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.660 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.660 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.661 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.661 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.661 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.661 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.661 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945611.768768_228.1", 2020-11-09T18:15:52.661 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.661 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-39:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.39", 2020-11-09T18:15:52.661 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:31.768768717Z", 2020-11-09T18:15:52.662 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.662 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.662 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.662 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.662 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.662 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.662 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_DoCr53MUjfmeA1PTWsLAoZg", 2020-11-09T18:15:52.663 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.663 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.663 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.663 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.663 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.663 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.663 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.664 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.664 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945611.771007_229.1", 2020-11-09T18:15:52.664 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.664 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-39", 2020-11-09T18:15:52.664 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:31.771007696Z", 2020-11-09T18:15:52.664 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.664 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.664 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.665 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.665 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.665 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.665 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_pGgwgGXPCFeV9dl7j-mkG2D", 2020-11-09T18:15:52.665 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.665 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.665 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.666 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.666 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.666 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.666 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.666 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.666 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945611.776767_230.1", 2020-11-09T18:15:52.666 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.666 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-39", 2020-11-09T18:15:52.667 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:31.776767187Z", 2020-11-09T18:15:52.667 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.667 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.667 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.667 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.667 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.667 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.668 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_pGgwgGXPCFeV9dl7j-mkG2D", 2020-11-09T18:15:52.668 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.668 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.668 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.668 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.668 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.668 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.668 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.669 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.669 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945612.536576_231.1", 2020-11-09T18:15:52.669 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.669 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-40:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.40", 2020-11-09T18:15:52.669 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:32.536576361Z", 2020-11-09T18:15:52.669 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.669 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.670 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.670 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.670 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.670 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.670 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_eRiB6BdOLC_U7ajfFCuPyzq", 2020-11-09T18:15:52.670 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.670 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.671 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.671 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.671 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.671 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.671 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.671 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.671 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945612.542749_232.1", 2020-11-09T18:15:52.672 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.672 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-40:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.40", 2020-11-09T18:15:52.672 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:32.542749842Z", 2020-11-09T18:15:52.672 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.672 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.672 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.672 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.673 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.673 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.673 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_eRiB6BdOLC_U7ajfFCuPyzq", 2020-11-09T18:15:52.673 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.673 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.673 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.673 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.673 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.674 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.674 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.674 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.674 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945612.545398_233.1", 2020-11-09T18:15:52.674 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.674 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-40", 2020-11-09T18:15:52.674 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:32.545398453Z", 2020-11-09T18:15:52.675 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.675 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.675 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.675 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.675 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.675 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.675 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_v-PHMVkQGnPvIMVgQXzEv-e", 2020-11-09T18:15:52.676 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.676 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.676 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.676 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.676 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.676 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.676 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.676 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.677 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945612.550287_234.1", 2020-11-09T18:15:52.677 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.677 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-40", 2020-11-09T18:15:52.677 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:32.550287369Z", 2020-11-09T18:15:52.677 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.677 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.677 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.678 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.678 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.678 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.678 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_v-PHMVkQGnPvIMVgQXzEv-e", 2020-11-09T18:15:52.678 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.678 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.678 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.679 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.679 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.679 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.679 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.679 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.679 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945612.641557_236.1", 2020-11-09T18:15:52.680 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.681 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-41:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.41", 2020-11-09T18:15:52.681 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:32.641557244Z", 2020-11-09T18:15:52.681 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.682 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.682 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.682 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.682 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.682 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.682 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_VbEzzl6PqzRKUgwPXZGxMNK", 2020-11-09T18:15:52.682 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.683 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.683 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.683 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.683 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.683 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.683 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.683 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.684 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945612.646933_237.1", 2020-11-09T18:15:52.684 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.684 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-41:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.41", 2020-11-09T18:15:52.684 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:32.646933508Z", 2020-11-09T18:15:52.684 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.684 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.684 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.684 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.685 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.685 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.685 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_VbEzzl6PqzRKUgwPXZGxMNK", 2020-11-09T18:15:52.685 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.685 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.685 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.685 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.686 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.686 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.686 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.686 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.686 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945612.649001_238.1", 2020-11-09T18:15:52.686 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.686 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-41", 2020-11-09T18:15:52.687 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:32.649001607Z", 2020-11-09T18:15:52.687 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.687 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.687 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.687 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.687 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.687 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.687 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_-QlJbrN7eIOTl-coxNoWB0_", 2020-11-09T18:15:52.688 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.688 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.688 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.688 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.688 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.688 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.688 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.689 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.689 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945612.653005_239.1", 2020-11-09T18:15:52.689 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.689 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-41", 2020-11-09T18:15:52.689 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:32.653005302Z", 2020-11-09T18:15:52.689 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.689 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.690 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.690 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.690 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.690 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.690 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_-QlJbrN7eIOTl-coxNoWB0_", 2020-11-09T18:15:52.690 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.690 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.691 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.691 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.691 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.691 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.691 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.691 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.691 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945563.557787_47.1", 2020-11-09T18:15:52.691 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.692 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-1:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.1", 2020-11-09T18:15:52.692 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:43.557787352Z", 2020-11-09T18:15:52.692 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.692 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.692 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.692 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.692 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.693 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.693 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_5-8ThY0rD5nJEgi4q-i6qW8", 2020-11-09T18:15:52.693 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.693 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.693 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.693 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.693 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.694 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.694 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.694 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.694 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945563.567624_51.1", 2020-11-09T18:15:52.694 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.694 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-1:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.1", 2020-11-09T18:15:52.694 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:43.567624280Z", 2020-11-09T18:15:52.695 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.695 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.695 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.695 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.695 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.695 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.695 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_5-8ThY0rD5nJEgi4q-i6qW8", 2020-11-09T18:15:52.696 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.696 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.696 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.696 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.696 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.696 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.696 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.696 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.697 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945563.569310_52.1", 2020-11-09T18:15:52.697 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.697 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-1", 2020-11-09T18:15:52.697 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:43.569310797Z", 2020-11-09T18:15:52.697 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.697 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.697 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.698 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.698 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.698 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.698 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_KrRBCosiXlBz0ajusgkrpWS", 2020-11-09T18:15:52.698 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.698 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.698 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.699 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.699 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.699 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.699 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.699 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.699 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945563.577890_53.1", 2020-11-09T18:15:52.699 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.699 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-1", 2020-11-09T18:15:52.700 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:43.577890725Z", 2020-11-09T18:15:52.700 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.700 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.700 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.700 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.700 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.700 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.701 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_KrRBCosiXlBz0ajusgkrpWS", 2020-11-09T18:15:52.701 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.701 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.701 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.701 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.701 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.701 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.701 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.702 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.702 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945563.608973_54.1", 2020-11-09T18:15:52.702 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.702 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-2:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.2", 2020-11-09T18:15:52.702 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:43.608973238Z", 2020-11-09T18:15:52.702 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.702 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.703 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.703 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.703 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.703 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.703 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_eZ3F6MzV7ES6qOIMaFzRrOO", 2020-11-09T18:15:52.703 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.703 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.704 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.704 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.704 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.704 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.704 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.704 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.704 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945563.618816_55.1", 2020-11-09T18:15:52.704 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.705 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-2:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.2", 2020-11-09T18:15:52.705 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:43.618816140Z", 2020-11-09T18:15:52.705 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.705 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.705 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.705 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.706 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.706 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.706 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_eZ3F6MzV7ES6qOIMaFzRrOO", 2020-11-09T18:15:52.706 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.706 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.706 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.706 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.707 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.707 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.707 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.707 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.707 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945563.621200_56.1", 2020-11-09T18:15:52.707 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.707 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-2", 2020-11-09T18:15:52.708 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:43.621200541Z", 2020-11-09T18:15:52.708 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.708 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.708 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.708 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.708 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.708 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.708 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_rj9KSIrvtNvXYWU2UYD1Z_m", 2020-11-09T18:15:52.709 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.709 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.709 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.709 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.709 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.709 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.709 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.710 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.710 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945563.627302_57.1", 2020-11-09T18:15:52.710 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.710 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-2", 2020-11-09T18:15:52.710 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:43.627302782Z", 2020-11-09T18:15:52.710 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.710 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.711 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.711 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.711 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.711 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.711 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_rj9KSIrvtNvXYWU2UYD1Z_m", 2020-11-09T18:15:52.711 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.711 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.711 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.712 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.712 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.712 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.712 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.712 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.712 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945567.627267_58.1", 2020-11-09T18:15:52.712 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.713 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-3:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.3", 2020-11-09T18:15:52.713 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:47.627267860Z", 2020-11-09T18:15:52.713 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.713 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.713 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.713 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.713 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.714 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.714 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_TnwqP2_mu3KZnFJOXhnO-DD", 2020-11-09T18:15:52.714 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.714 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.714 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.714 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.714 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.714 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.715 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.715 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.715 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945567.630834_59.1", 2020-11-09T18:15:52.715 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.715 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-3:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.3", 2020-11-09T18:15:52.715 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:47.630834733Z", 2020-11-09T18:15:52.715 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.716 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.716 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.716 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.716 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.716 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.716 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_TnwqP2_mu3KZnFJOXhnO-DD", 2020-11-09T18:15:52.716 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.716 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.717 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.717 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.717 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.717 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.717 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.717 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.717 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945567.632589_60.1", 2020-11-09T18:15:52.718 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.718 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-3", 2020-11-09T18:15:52.718 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:47.632589277Z", 2020-11-09T18:15:52.718 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.718 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.718 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.718 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.719 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.719 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.719 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_MHzcuL8U9yY5bsA8iZvWhTc", 2020-11-09T18:15:52.719 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.719 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.719 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.719 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.720 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.720 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.720 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.720 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.720 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945567.636634_61.1", 2020-11-09T18:15:52.720 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.720 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-3", 2020-11-09T18:15:52.721 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:47.636634661Z", 2020-11-09T18:15:52.721 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.721 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.721 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.721 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.721 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.721 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.721 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_MHzcuL8U9yY5bsA8iZvWhTc", 2020-11-09T18:15:52.722 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.722 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.722 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.722 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.722 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.722 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.722 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.723 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.723 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945570.370676_62.1", 2020-11-09T18:15:52.723 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.723 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-4:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.4", 2020-11-09T18:15:52.723 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:50.370676662Z", 2020-11-09T18:15:52.723 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.724 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.724 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.724 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.724 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.724 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.724 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_ufcC505t7ROkSzsmsL1irho", 2020-11-09T18:15:52.724 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.725 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.725 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.725 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.725 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.725 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.725 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.725 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.725 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945570.375714_64.1", 2020-11-09T18:15:52.726 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.726 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-4:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.4", 2020-11-09T18:15:52.726 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:50.375714519Z", 2020-11-09T18:15:52.726 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.726 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.726 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.726 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.727 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.727 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.727 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_ufcC505t7ROkSzsmsL1irho", 2020-11-09T18:15:52.727 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.727 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.727 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.727 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.728 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.728 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.728 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.728 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.728 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945570.377850_65.1", 2020-11-09T18:15:52.728 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.728 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-4", 2020-11-09T18:15:52.729 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:50.377850670Z", 2020-11-09T18:15:52.729 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.729 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.729 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.729 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.729 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.729 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.729 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_xnR_YIJ8zUxxn61yz_DTUkk", 2020-11-09T18:15:52.730 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.730 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.730 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.730 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.730 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.730 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.730 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.731 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.731 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945570.382459_66.1", 2020-11-09T18:15:52.731 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.731 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-4", 2020-11-09T18:15:52.731 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:50.382459732Z", 2020-11-09T18:15:52.731 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.731 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.732 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.732 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.732 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.732 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.732 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_xnR_YIJ8zUxxn61yz_DTUkk", 2020-11-09T18:15:52.732 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.732 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.732 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.733 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.733 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.733 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.733 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.733 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.733 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945570.406070_67.1", 2020-11-09T18:15:52.733 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.734 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-5:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.5", 2020-11-09T18:15:52.734 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:50.406070925Z", 2020-11-09T18:15:52.734 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.734 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.734 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.734 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.734 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.735 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.735 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_UShZGGBB4kBaxRzXtVmZBUW", 2020-11-09T18:15:52.735 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.735 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.735 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.735 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.735 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.736 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.736 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.736 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.736 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945570.410945_68.1", 2020-11-09T18:15:52.736 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.736 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-5:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.5", 2020-11-09T18:15:52.736 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:50.410945255Z", 2020-11-09T18:15:52.736 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.737 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.737 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.737 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.737 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.737 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.737 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_UShZGGBB4kBaxRzXtVmZBUW", 2020-11-09T18:15:52.737 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.738 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.738 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.738 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.738 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.738 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.738 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.738 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.738 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945570.412796_69.1", 2020-11-09T18:15:52.739 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.739 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-5", 2020-11-09T18:15:52.739 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:50.412796913Z", 2020-11-09T18:15:52.739 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.739 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.739 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.739 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.740 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.740 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.740 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_5bx-BH8ijLijK0Hb5tRXntZ", 2020-11-09T18:15:52.740 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.740 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.740 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.740 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.740 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.741 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.741 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.741 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.741 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945570.416832_70.1", 2020-11-09T18:15:52.741 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.741 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-5", 2020-11-09T18:15:52.741 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:50.416832729Z", 2020-11-09T18:15:52.742 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.742 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.742 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.742 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.742 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.742 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.742 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_5bx-BH8ijLijK0Hb5tRXntZ", 2020-11-09T18:15:52.743 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.743 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.743 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.743 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.743 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.743 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.743 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.743 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.744 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945570.461105_71.1", 2020-11-09T18:15:52.744 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.744 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-6:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.6", 2020-11-09T18:15:52.744 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:50.461105755Z", 2020-11-09T18:15:52.744 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.744 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.744 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.744 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.745 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.745 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.745 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_qWlHsdsWYJHc0mQkDC3vzYx", 2020-11-09T18:15:52.745 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.745 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.745 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.745 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.746 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.746 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.746 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.746 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.746 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945570.473537_72.1", 2020-11-09T18:15:52.746 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.746 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-6:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.6", 2020-11-09T18:15:52.747 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:50.473537938Z", 2020-11-09T18:15:52.747 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.747 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.747 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.747 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.747 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.747 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.747 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_qWlHsdsWYJHc0mQkDC3vzYx", 2020-11-09T18:15:52.748 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.748 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.748 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.748 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.748 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.748 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.748 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.749 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.749 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945570.477860_73.1", 2020-11-09T18:15:52.749 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.749 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-6", 2020-11-09T18:15:52.749 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:50.477860682Z", 2020-11-09T18:15:52.749 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.749 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.749 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.750 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.750 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.750 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.750 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dmuyyeGa1O--zFbpSa0SA8D", 2020-11-09T18:15:52.750 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.750 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.750 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.751 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.751 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.751 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.751 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.751 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.751 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945570.489122_74.1", 2020-11-09T18:15:52.751 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.751 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-6", 2020-11-09T18:15:52.752 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:50.489122994Z", 2020-11-09T18:15:52.752 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.752 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.752 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.752 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.752 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.752 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.753 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_dmuyyeGa1O--zFbpSa0SA8D", 2020-11-09T18:15:52.753 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.753 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.753 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.753 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.753 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.753 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.753 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.754 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.754 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945579.353942_75.1", 2020-11-09T18:15:52.754 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.754 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-7:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.7", 2020-11-09T18:15:52.754 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:59.353942489Z", 2020-11-09T18:15:52.754 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.754 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.755 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.755 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.755 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.755 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.755 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_GIkCDw3GWmDRO3-A0auRtxB", 2020-11-09T18:15:52.755 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.755 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.755 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.756 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.756 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.756 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.756 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.756 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.756 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945579.359041_78.1", 2020-11-09T18:15:52.756 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.757 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-7:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.7", 2020-11-09T18:15:52.757 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:59.359041450Z", 2020-11-09T18:15:52.757 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.757 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.757 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.757 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.757 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.757 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.758 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_GIkCDw3GWmDRO3-A0auRtxB", 2020-11-09T18:15:52.758 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.758 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.758 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.758 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.758 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.758 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.759 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.759 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.759 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945579.361526_79.1", 2020-11-09T18:15:52.759 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.759 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-7", 2020-11-09T18:15:52.759 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:59.361526900Z", 2020-11-09T18:15:52.759 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.760 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.760 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.760 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.760 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.760 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.760 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_euurhSSx6WmV693QUVvezHc", 2020-11-09T18:15:52.760 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.760 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.761 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.761 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.761 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.761 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.761 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.761 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.762 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945579.367322_80.1", 2020-11-09T18:15:52.762 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.762 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-7", 2020-11-09T18:15:52.762 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:59.367322143Z", 2020-11-09T18:15:52.762 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.762 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.763 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.763 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.763 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.763 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.763 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_euurhSSx6WmV693QUVvezHc", 2020-11-09T18:15:52.763 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.764 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.764 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.764 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.764 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.764 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.764 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.765 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.765 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945579.395388_81.1", 2020-11-09T18:15:52.765 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.765 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-8:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.8", 2020-11-09T18:15:52.765 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:59.395388143Z", 2020-11-09T18:15:52.766 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.766 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.766 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.766 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.767 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.767 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.767 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_yBzYlhlGW2rcaJy91e-7eds", 2020-11-09T18:15:52.767 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.767 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.767 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.768 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.768 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.768 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.768 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.768 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.768 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945579.400026_82.1", 2020-11-09T18:15:52.768 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.768 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-8:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.8", 2020-11-09T18:15:52.769 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:59.400026920Z", 2020-11-09T18:15:52.769 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.769 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.769 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.769 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.769 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.769 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.770 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_yBzYlhlGW2rcaJy91e-7eds", 2020-11-09T18:15:52.770 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.770 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.770 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.770 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.770 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.770 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.771 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.771 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.771 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945579.402183_83.1", 2020-11-09T18:15:52.771 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.771 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-8", 2020-11-09T18:15:52.771 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:59.402183037Z", 2020-11-09T18:15:52.771 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.771 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.772 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.772 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.772 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.772 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.772 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_OGdV9KW-t2xB2NYjc8pNB2U", 2020-11-09T18:15:52.772 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.772 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.773 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.773 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.773 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.773 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.773 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.773 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.773 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945579.406495_84.1", 2020-11-09T18:15:52.773 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.774 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-8", 2020-11-09T18:15:52.774 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:59.406495328Z", 2020-11-09T18:15:52.774 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.774 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.774 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.774 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.774 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.775 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.775 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_OGdV9KW-t2xB2NYjc8pNB2U", 2020-11-09T18:15:52.775 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.775 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.775 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.775 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.775 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.776 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.776 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.776 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.776 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945579.450093_85.1", 2020-11-09T18:15:52.776 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.776 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-9:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.9", 2020-11-09T18:15:52.776 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:59.450093555Z", 2020-11-09T18:15:52.776 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.777 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.777 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.777 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.777 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.777 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.777 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_sx-mxXXsFhATfbPuJXuh1It", 2020-11-09T18:15:52.777 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.778 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.778 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.778 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.778 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.778 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.778 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.778 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.778 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945579.455372_86.1", 2020-11-09T18:15:52.779 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:52.779 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-9:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.9", 2020-11-09T18:15:52.779 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:59.455372433Z", 2020-11-09T18:15:52.779 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.779 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.779 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.779 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.780 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.780 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.780 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_sx-mxXXsFhATfbPuJXuh1It", 2020-11-09T18:15:52.780 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.780 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.780 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.780 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.780 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.781 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.781 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.781 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.781 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945579.459352_87.1", 2020-11-09T18:15:52.781 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.781 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-9", 2020-11-09T18:15:52.781 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:59.459352980Z", 2020-11-09T18:15:52.782 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.782 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.782 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.782 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.782 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.782 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.782 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_CehTzP8PVTogocbVBJZMUE-", 2020-11-09T18:15:52.783 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.783 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.783 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.783 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:52.783 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.783 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.783 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.783 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.784 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945579.467305_88.1", 2020-11-09T18:15:52.784 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.784 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-9", 2020-11-09T18:15:52.784 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:12:59.467305227Z", 2020-11-09T18:15:52.784 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.784 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.784 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:52.785 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:52.785 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.785 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.785 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_CehTzP8PVTogocbVBJZMUE-", 2020-11-09T18:15:52.785 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.785 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.785 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.785 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.786 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.786 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.786 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.786 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.786 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945582.240772_89.1", 2020-11-09T18:15:52.786 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.786 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-7", 2020-11-09T18:15:52.787 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:02.240772877Z", 2020-11-09T18:15:52.787 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.787 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.787 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_euurhSSx6WmV693QUVvezHc", 2020-11-09T18:15:52.787 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.787 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.787 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.788 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_euurhSSx6WmV693QUVvezHc", 2020-11-09T18:15:52.788 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.788 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.788 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.788 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "remove" 2020-11-09T18:15:52.788 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.788 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.788 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.789 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.789 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945582.251051_91.1", 2020-11-09T18:15:52.789 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.789 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-7", 2020-11-09T18:15:52.789 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:02.251051719Z", 2020-11-09T18:15:52.789 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.789 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.790 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_euurhSSx6WmV693QUVvezHc", 2020-11-09T18:15:52.790 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.790 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.790 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.790 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_euurhSSx6WmV693QUVvezHc", 2020-11-09T18:15:52.790 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.790 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.790 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.791 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.791 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.791 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.791 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.791 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.791 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945582.313479_92.1", 2020-11-09T18:15:52.791 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.792 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-8", 2020-11-09T18:15:52.792 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:02.313479625Z", 2020-11-09T18:15:52.792 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.792 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.792 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_OGdV9KW-t2xB2NYjc8pNB2U", 2020-11-09T18:15:52.792 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.792 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.792 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.793 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_OGdV9KW-t2xB2NYjc8pNB2U", 2020-11-09T18:15:52.793 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.793 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.793 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.793 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "remove" 2020-11-09T18:15:52.793 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.793 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.794 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.794 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.794 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945582.321374_93.1", 2020-11-09T18:15:52.794 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.794 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-8", 2020-11-09T18:15:52.794 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:02.321374298Z", 2020-11-09T18:15:52.794 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.795 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.795 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_OGdV9KW-t2xB2NYjc8pNB2U", 2020-11-09T18:15:52.795 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.795 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.795 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.795 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_OGdV9KW-t2xB2NYjc8pNB2U", 2020-11-09T18:15:52.795 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.795 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.796 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.796 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.796 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.796 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.796 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.796 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.796 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945582.425686_94.1", 2020-11-09T18:15:52.797 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.797 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-9", 2020-11-09T18:15:52.797 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:02.425686180Z", 2020-11-09T18:15:52.797 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.797 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.797 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_CehTzP8PVTogocbVBJZMUE-", 2020-11-09T18:15:52.797 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.797 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.798 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.798 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_CehTzP8PVTogocbVBJZMUE-", 2020-11-09T18:15:52.798 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.798 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.798 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.798 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "remove" 2020-11-09T18:15:52.798 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.799 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.799 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.799 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:52.799 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945582.432939_95.1", 2020-11-09T18:15:52.799 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:52.799 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-9", 2020-11-09T18:15:52.799 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:02.432939430Z", 2020-11-09T18:15:52.800 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:52.800 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:52.800 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_CehTzP8PVTogocbVBJZMUE-", 2020-11-09T18:15:52.800 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:52.800 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.800 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:52.800 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_CehTzP8PVTogocbVBJZMUE-", 2020-11-09T18:15:52.800 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 2 2020-11-09T18:15:52.801 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:52.801 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:52.801 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:52.801 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.801 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.801 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:52.801 INFO:teuthology.orchestra.run.smithi051.stdout:] 2020-11-09T18:15:52.805 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:52.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:52.806 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:15:52.844 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:15:52.845 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:15:53.078 INFO:teuthology.orchestra.run.smithi078.stdout:[ 2020-11-09T18:15:53.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.078 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945439.484659_48.1", 2020-11-09T18:15:53.078 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "user", 2020-11-09T18:15:53.078 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rgw-multisite-test-user", 2020-11-09T18:15:53.079 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:10:39.484659067Z", 2020-11-09T18:15:53.079 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.079 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.079 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.079 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.079 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.080 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_qxUmJfyn2gLxVOOHvE_t0VQ", 2020-11-09T18:15:53.080 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.080 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.080 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.080 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.080 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.081 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945439.492888_50.1", 2020-11-09T18:15:53.081 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "user", 2020-11-09T18:15:53.081 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rgw-multisite-test-user", 2020-11-09T18:15:53.081 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:10:39.492888346Z", 2020-11-09T18:15:53.081 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.082 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.082 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.082 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.082 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.082 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_qxUmJfyn2gLxVOOHvE_t0VQ", 2020-11-09T18:15:53.082 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.083 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.084 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.084 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.084 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.085 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945586.256023_51.1", 2020-11-09T18:15:53.086 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.086 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-10:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.10", 2020-11-09T18:15:53.086 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:06.256023190Z", 2020-11-09T18:15:53.087 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.087 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.087 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.087 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.088 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.088 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_xk1WELKUHZIqmUd5oTGWrEo", 2020-11-09T18:15:53.088 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.088 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.089 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.089 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.089 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.089 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945586.261184_59.1", 2020-11-09T18:15:53.090 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.090 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-10:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.10", 2020-11-09T18:15:53.090 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:06.261184780Z", 2020-11-09T18:15:53.090 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.090 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.091 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.091 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.091 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.091 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_xk1WELKUHZIqmUd5oTGWrEo", 2020-11-09T18:15:53.091 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.092 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.092 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.092 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.092 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.092 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945586.263844_60.1", 2020-11-09T18:15:53.092 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.093 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-10", 2020-11-09T18:15:53.093 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:06.263844660Z", 2020-11-09T18:15:53.093 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.093 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.093 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.093 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.094 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.094 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_p-7-bX-sO2hh9J_2Nonu8rp", 2020-11-09T18:15:53.094 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.094 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.094 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.095 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.095 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.095 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945586.268581_61.1", 2020-11-09T18:15:53.095 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.095 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-10", 2020-11-09T18:15:53.096 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:06.268581555Z", 2020-11-09T18:15:53.096 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.096 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.096 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.096 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.096 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.096 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_p-7-bX-sO2hh9J_2Nonu8rp", 2020-11-09T18:15:53.097 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.097 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.097 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.097 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.097 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.098 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945587.202762_62.1", 2020-11-09T18:15:53.098 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.098 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-11:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.11", 2020-11-09T18:15:53.098 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:07.202762446Z", 2020-11-09T18:15:53.098 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.098 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.099 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.099 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.099 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.099 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_5opGpz-RxCA9co32z-sLP0e", 2020-11-09T18:15:53.099 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.099 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.100 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.100 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.100 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.100 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945587.207905_63.1", 2020-11-09T18:15:53.100 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.101 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-11:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.11", 2020-11-09T18:15:53.101 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:07.207905728Z", 2020-11-09T18:15:53.101 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.101 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.101 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.102 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.102 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.102 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_5opGpz-RxCA9co32z-sLP0e", 2020-11-09T18:15:53.102 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.102 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.103 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.103 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.103 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.103 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945587.210891_64.1", 2020-11-09T18:15:53.103 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.103 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-11", 2020-11-09T18:15:53.104 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:07.210891570Z", 2020-11-09T18:15:53.104 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.104 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.104 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.104 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.104 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.105 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_QiNkPq8a5F1hoeuAtqCxHNQ", 2020-11-09T18:15:53.105 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.105 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.105 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.105 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.105 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.106 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945587.216871_65.1", 2020-11-09T18:15:53.106 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.106 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-11", 2020-11-09T18:15:53.106 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:07.216871948Z", 2020-11-09T18:15:53.106 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.106 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.107 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.107 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.107 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.107 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_QiNkPq8a5F1hoeuAtqCxHNQ", 2020-11-09T18:15:53.107 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.108 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.108 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.108 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.108 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.109 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945588.126275_66.1", 2020-11-09T18:15:53.109 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.109 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-12:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.12", 2020-11-09T18:15:53.109 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:08.126275141Z", 2020-11-09T18:15:53.109 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.109 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.110 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.110 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.110 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.110 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_7WQjdW6VYC3fP3T1DUBlsCk", 2020-11-09T18:15:53.110 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.111 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.111 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.111 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.111 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.111 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945588.133515_67.1", 2020-11-09T18:15:53.112 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.112 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-12:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.12", 2020-11-09T18:15:53.112 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:08.133515560Z", 2020-11-09T18:15:53.112 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.112 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.112 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.112 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.113 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.113 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_7WQjdW6VYC3fP3T1DUBlsCk", 2020-11-09T18:15:53.113 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.113 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.113 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.114 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.114 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.114 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945588.135642_68.1", 2020-11-09T18:15:53.114 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.114 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-12", 2020-11-09T18:15:53.114 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:08.135642294Z", 2020-11-09T18:15:53.115 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.115 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.115 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.115 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.116 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.116 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_PdQ7asc_0k_3_XotzK7eatQ", 2020-11-09T18:15:53.116 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.117 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.117 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.117 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.117 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.118 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945588.143746_69.1", 2020-11-09T18:15:53.118 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.118 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-12", 2020-11-09T18:15:53.118 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:08.143746572Z", 2020-11-09T18:15:53.118 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.118 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.118 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.119 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.119 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.119 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_PdQ7asc_0k_3_XotzK7eatQ", 2020-11-09T18:15:53.119 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.119 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.119 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.120 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.120 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.120 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945588.381357_70.1", 2020-11-09T18:15:53.120 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.120 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-13:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.13", 2020-11-09T18:15:53.121 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:08.381357113Z", 2020-11-09T18:15:53.121 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.121 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.121 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.121 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.121 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.122 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_jRsLDHCkxygm_NRmusaemxh", 2020-11-09T18:15:53.122 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.122 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.122 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.122 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.122 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.123 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945588.386292_71.1", 2020-11-09T18:15:53.123 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.123 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-13:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.13", 2020-11-09T18:15:53.123 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:08.386292596Z", 2020-11-09T18:15:53.123 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.123 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.124 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.124 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.124 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.124 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_jRsLDHCkxygm_NRmusaemxh", 2020-11-09T18:15:53.124 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.125 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.125 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.125 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.125 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.125 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945588.391790_72.1", 2020-11-09T18:15:53.126 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.126 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-13", 2020-11-09T18:15:53.126 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:08.391790067Z", 2020-11-09T18:15:53.126 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.126 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.126 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.126 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.127 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.127 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_1zAxS7Eu4GCsgwIw7eCrrnz", 2020-11-09T18:15:53.127 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.127 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.127 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.127 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.128 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.128 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945588.397490_73.1", 2020-11-09T18:15:53.128 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.128 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-13", 2020-11-09T18:15:53.128 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:08.397490757Z", 2020-11-09T18:15:53.129 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.129 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.129 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.129 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.129 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.129 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_1zAxS7Eu4GCsgwIw7eCrrnz", 2020-11-09T18:15:53.129 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.130 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.130 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.130 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.130 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.131 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945589.380394_74.1", 2020-11-09T18:15:53.131 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.131 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-14:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.14", 2020-11-09T18:15:53.131 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:09.380394647Z", 2020-11-09T18:15:53.131 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.131 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.132 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.132 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.132 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.132 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_ZoR7I5lOnVtZkMZ3aHO24Vp", 2020-11-09T18:15:53.132 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.133 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.133 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.133 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.133 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.134 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945589.385117_75.1", 2020-11-09T18:15:53.134 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.134 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-14:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.14", 2020-11-09T18:15:53.134 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:09.385117237Z", 2020-11-09T18:15:53.134 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.134 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.134 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.135 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.135 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.135 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_ZoR7I5lOnVtZkMZ3aHO24Vp", 2020-11-09T18:15:53.135 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.135 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.135 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.136 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.136 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.136 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945589.387155_76.1", 2020-11-09T18:15:53.136 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.136 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-14", 2020-11-09T18:15:53.138 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:09.387155581Z", 2020-11-09T18:15:53.138 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.138 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.138 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.138 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.138 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.139 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_iQYh1hCLyur_paDmK5MzSiw", 2020-11-09T18:15:53.139 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.139 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.139 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.139 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.139 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.140 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945589.391251_77.1", 2020-11-09T18:15:53.140 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.140 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-14", 2020-11-09T18:15:53.140 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:09.391251540Z", 2020-11-09T18:15:53.140 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.141 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.141 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.141 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.141 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.141 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_iQYh1hCLyur_paDmK5MzSiw", 2020-11-09T18:15:53.141 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.142 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.142 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.142 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.142 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.142 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.142 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945589.523751_78.1", 2020-11-09T18:15:53.143 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.143 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-15:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.15", 2020-11-09T18:15:53.143 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:09.523751440Z", 2020-11-09T18:15:53.143 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.143 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.143 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.143 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.144 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.144 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_KVNlDu4zRBEGnM1a31uWniG", 2020-11-09T18:15:53.144 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.144 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.144 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.144 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.145 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.145 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945589.534160_79.1", 2020-11-09T18:15:53.145 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.145 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-15:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.15", 2020-11-09T18:15:53.145 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:09.534160171Z", 2020-11-09T18:15:53.146 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.146 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.146 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.146 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.146 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.146 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_KVNlDu4zRBEGnM1a31uWniG", 2020-11-09T18:15:53.147 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.147 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.147 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.147 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.147 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.147 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.148 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945589.536862_80.1", 2020-11-09T18:15:53.148 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.148 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-15", 2020-11-09T18:15:53.148 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:09.536862933Z", 2020-11-09T18:15:53.148 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.148 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.148 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.149 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.149 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.149 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_Cq6C5mA4ASiylyTIC5SMBlP", 2020-11-09T18:15:53.149 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.149 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.149 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.150 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.150 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.150 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945589.541920_81.1", 2020-11-09T18:15:53.150 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.150 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-15", 2020-11-09T18:15:53.151 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:09.541920734Z", 2020-11-09T18:15:53.151 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.151 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.151 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.151 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.151 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.151 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_Cq6C5mA4ASiylyTIC5SMBlP", 2020-11-09T18:15:53.152 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.152 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.152 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.152 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.152 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.153 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945590.011767_82.1", 2020-11-09T18:15:53.153 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.153 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-16:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.16", 2020-11-09T18:15:53.153 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:10.011767329Z", 2020-11-09T18:15:53.153 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.153 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.153 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.154 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.154 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.154 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_fJsd-7NZirDCEodHfdpY9oA", 2020-11-09T18:15:53.154 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.154 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.155 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.155 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.155 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.155 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.155 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945590.016722_83.1", 2020-11-09T18:15:53.155 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.156 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-16:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.16", 2020-11-09T18:15:53.156 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:10.016722284Z", 2020-11-09T18:15:53.156 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.156 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.156 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.156 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.156 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.157 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_fJsd-7NZirDCEodHfdpY9oA", 2020-11-09T18:15:53.157 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.157 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.157 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.157 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.157 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.158 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.158 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945590.019556_84.1", 2020-11-09T18:15:53.158 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.158 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-16", 2020-11-09T18:15:53.158 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:10.019556366Z", 2020-11-09T18:15:53.158 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.158 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.159 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.159 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.159 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.159 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_83VseKNDEzwg2zXJyAxVSNV", 2020-11-09T18:15:53.159 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.159 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.160 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.160 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.160 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.160 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945590.028620_85.1", 2020-11-09T18:15:53.160 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.161 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-16", 2020-11-09T18:15:53.161 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:10.028620030Z", 2020-11-09T18:15:53.161 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.161 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.161 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.161 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.162 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.162 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_83VseKNDEzwg2zXJyAxVSNV", 2020-11-09T18:15:53.162 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.162 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.162 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.162 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.162 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.163 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.163 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945590.059189_86.1", 2020-11-09T18:15:53.163 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.163 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-17:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.17", 2020-11-09T18:15:53.163 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:10.059189312Z", 2020-11-09T18:15:53.163 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.164 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.164 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.164 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.164 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.164 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_nhpbrE0BaTecbpBGZZxkC_9", 2020-11-09T18:15:53.164 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.165 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.165 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.165 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.165 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.165 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.165 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945590.065317_87.1", 2020-11-09T18:15:53.166 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.166 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-17:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.17", 2020-11-09T18:15:53.166 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:10.065317701Z", 2020-11-09T18:15:53.166 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.166 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.166 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.166 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.167 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.167 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_nhpbrE0BaTecbpBGZZxkC_9", 2020-11-09T18:15:53.167 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.167 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.167 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.167 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.168 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.168 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945590.068336_88.1", 2020-11-09T18:15:53.168 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.168 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-17", 2020-11-09T18:15:53.168 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:10.068336838Z", 2020-11-09T18:15:53.169 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.169 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.169 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.169 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.169 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.169 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_DZ4ozYh7iBiXpSqfO2MFhvn", 2020-11-09T18:15:53.169 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.170 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.170 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.170 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.170 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.171 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945590.076749_89.1", 2020-11-09T18:15:53.171 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.171 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-17", 2020-11-09T18:15:53.171 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:10.076749917Z", 2020-11-09T18:15:53.171 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.171 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.171 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.172 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.172 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.172 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_DZ4ozYh7iBiXpSqfO2MFhvn", 2020-11-09T18:15:53.172 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.172 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.172 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.173 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.173 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.173 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945590.120726_90.1", 2020-11-09T18:15:53.173 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.173 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-18:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.18", 2020-11-09T18:15:53.174 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:10.120726234Z", 2020-11-09T18:15:53.174 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.174 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.174 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.174 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.174 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.174 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_R8b8xLVdGNeTGDdC42EuEfE", 2020-11-09T18:15:53.175 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.175 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.175 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.175 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.175 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.176 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945590.127213_91.1", 2020-11-09T18:15:53.176 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.176 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-18:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.18", 2020-11-09T18:15:53.176 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:10.127213093Z", 2020-11-09T18:15:53.176 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.176 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.176 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.177 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.177 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.177 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_R8b8xLVdGNeTGDdC42EuEfE", 2020-11-09T18:15:53.177 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.177 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.178 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.178 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.178 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.178 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.178 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945590.129116_92.1", 2020-11-09T18:15:53.178 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.178 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-18", 2020-11-09T18:15:53.179 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:10.129116612Z", 2020-11-09T18:15:53.179 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.179 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.179 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.179 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.179 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.180 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_jvrbBsWxGWFBmAxvo-GL8oI", 2020-11-09T18:15:53.180 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.180 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.180 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.180 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.180 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.181 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945590.133065_93.1", 2020-11-09T18:15:53.181 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.181 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-18", 2020-11-09T18:15:53.181 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:10.133065786Z", 2020-11-09T18:15:53.181 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.181 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.182 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.182 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.182 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.182 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_jvrbBsWxGWFBmAxvo-GL8oI", 2020-11-09T18:15:53.182 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.183 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.183 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.183 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.183 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.183 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.183 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945590.156864_94.1", 2020-11-09T18:15:53.183 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.184 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-16:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.16", 2020-11-09T18:15:53.184 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:10.156864142Z", 2020-11-09T18:15:53.184 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.184 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.184 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_fJsd-7NZirDCEodHfdpY9oA", 2020-11-09T18:15:53.184 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.185 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.185 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_fJsd-7NZirDCEodHfdpY9oA", 2020-11-09T18:15:53.185 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.185 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.185 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.185 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.185 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.186 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945590.163596_95.1", 2020-11-09T18:15:53.186 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.186 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-16:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.16", 2020-11-09T18:15:53.186 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:10.163596031Z", 2020-11-09T18:15:53.186 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.187 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.187 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_fJsd-7NZirDCEodHfdpY9oA", 2020-11-09T18:15:53.187 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.187 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.187 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_fJsd-7NZirDCEodHfdpY9oA", 2020-11-09T18:15:53.187 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.188 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.188 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.188 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.188 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.188 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945590.168397_96.1", 2020-11-09T18:15:53.189 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.189 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-17:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.17", 2020-11-09T18:15:53.189 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:10.168397876Z", 2020-11-09T18:15:53.189 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.189 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.189 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_nhpbrE0BaTecbpBGZZxkC_9", 2020-11-09T18:15:53.189 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.190 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.190 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_nhpbrE0BaTecbpBGZZxkC_9", 2020-11-09T18:15:53.190 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.190 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.190 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.190 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.191 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.191 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945590.178873_97.1", 2020-11-09T18:15:53.191 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.191 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-17:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.17", 2020-11-09T18:15:53.191 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:10.178873937Z", 2020-11-09T18:15:53.192 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.192 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.192 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_nhpbrE0BaTecbpBGZZxkC_9", 2020-11-09T18:15:53.192 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.192 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.192 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_nhpbrE0BaTecbpBGZZxkC_9", 2020-11-09T18:15:53.193 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.193 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.193 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.193 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.193 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.194 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945590.190493_98.1", 2020-11-09T18:15:53.194 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.194 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-18:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.18", 2020-11-09T18:15:53.194 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:10.190493784Z", 2020-11-09T18:15:53.194 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.194 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.194 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_R8b8xLVdGNeTGDdC42EuEfE", 2020-11-09T18:15:53.195 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.195 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.195 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_R8b8xLVdGNeTGDdC42EuEfE", 2020-11-09T18:15:53.195 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.195 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.195 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.196 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.196 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.196 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945590.199376_99.1", 2020-11-09T18:15:53.196 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.196 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-18:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.18", 2020-11-09T18:15:53.197 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:10.199376961Z", 2020-11-09T18:15:53.197 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.197 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.197 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_R8b8xLVdGNeTGDdC42EuEfE", 2020-11-09T18:15:53.197 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.197 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.197 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_R8b8xLVdGNeTGDdC42EuEfE", 2020-11-09T18:15:53.198 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.198 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.198 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.198 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.198 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.199 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945592.270499_100.1", 2020-11-09T18:15:53.199 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.199 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-19:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.19", 2020-11-09T18:15:53.199 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:12.270499729Z", 2020-11-09T18:15:53.199 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.199 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.199 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.200 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.200 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.200 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_E6pXupKMFQLC75ImCoCkV74", 2020-11-09T18:15:53.200 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.200 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.201 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.201 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.201 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.201 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.201 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945592.275423_102.1", 2020-11-09T18:15:53.201 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.201 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-19:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.19", 2020-11-09T18:15:53.202 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:12.275423973Z", 2020-11-09T18:15:53.202 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.202 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.202 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.202 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.202 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.203 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_E6pXupKMFQLC75ImCoCkV74", 2020-11-09T18:15:53.203 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.203 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.203 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.203 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.203 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.204 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945592.278527_103.1", 2020-11-09T18:15:53.204 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.204 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-19", 2020-11-09T18:15:53.204 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:12.278527677Z", 2020-11-09T18:15:53.204 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.204 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.205 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.205 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.205 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.205 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_OGzY_hYHGuuTc6_1SixJvo6", 2020-11-09T18:15:53.205 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.205 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.206 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.206 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.206 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.206 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.206 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945592.284263_104.1", 2020-11-09T18:15:53.206 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.207 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-19", 2020-11-09T18:15:53.207 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:12.284263022Z", 2020-11-09T18:15:53.207 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.207 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.208 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.208 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.209 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.209 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_OGzY_hYHGuuTc6_1SixJvo6", 2020-11-09T18:15:53.209 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.209 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.209 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.209 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.210 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.210 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945592.291372_105.1", 2020-11-09T18:15:53.210 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.210 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-19:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.19", 2020-11-09T18:15:53.210 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:12.291372646Z", 2020-11-09T18:15:53.211 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.211 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.211 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_E6pXupKMFQLC75ImCoCkV74", 2020-11-09T18:15:53.211 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.211 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.211 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_E6pXupKMFQLC75ImCoCkV74", 2020-11-09T18:15:53.212 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.212 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.212 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.212 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.212 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.213 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945592.300193_106.1", 2020-11-09T18:15:53.213 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.213 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-19:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.19", 2020-11-09T18:15:53.213 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:12.300193728Z", 2020-11-09T18:15:53.213 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.213 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.213 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_E6pXupKMFQLC75ImCoCkV74", 2020-11-09T18:15:53.214 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.214 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.214 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_E6pXupKMFQLC75ImCoCkV74", 2020-11-09T18:15:53.214 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.214 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.215 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.215 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.215 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.215 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945594.039798_107.1", 2020-11-09T18:15:53.215 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.216 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-20:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.20", 2020-11-09T18:15:53.216 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:14.039798004Z", 2020-11-09T18:15:53.216 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.216 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.216 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.216 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.216 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.217 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_cfJ_IuSbXauFncnrlMQUrK1", 2020-11-09T18:15:53.217 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.217 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.217 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.217 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.217 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.218 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945594.043998_109.1", 2020-11-09T18:15:53.218 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.218 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-20:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.20", 2020-11-09T18:15:53.218 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:14.043998666Z", 2020-11-09T18:15:53.218 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.219 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.219 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.219 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.219 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.219 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_cfJ_IuSbXauFncnrlMQUrK1", 2020-11-09T18:15:53.219 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.220 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.220 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.220 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.220 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.220 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945594.046065_110.1", 2020-11-09T18:15:53.221 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.221 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-20", 2020-11-09T18:15:53.221 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:14.046065820Z", 2020-11-09T18:15:53.221 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.221 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.221 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.221 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.222 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.222 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_4-ey3kn91fmGef8EsodcNMf", 2020-11-09T18:15:53.222 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.222 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.222 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.222 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.223 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.223 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945594.050612_111.1", 2020-11-09T18:15:53.223 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.223 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-20", 2020-11-09T18:15:53.223 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:14.050612141Z", 2020-11-09T18:15:53.223 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.224 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.224 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.224 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.224 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.224 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_4-ey3kn91fmGef8EsodcNMf", 2020-11-09T18:15:53.224 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.225 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.225 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.225 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.225 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.225 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945595.459954_112.1", 2020-11-09T18:15:53.226 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.226 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-21:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.21", 2020-11-09T18:15:53.226 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:15.459954913Z", 2020-11-09T18:15:53.226 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.226 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.226 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.226 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.227 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.227 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_7weq2-_FEkT51PfhS1TopF2", 2020-11-09T18:15:53.227 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.227 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.227 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.227 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.228 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.228 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945595.464408_114.1", 2020-11-09T18:15:53.228 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.228 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-21:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.21", 2020-11-09T18:15:53.228 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:15.464408755Z", 2020-11-09T18:15:53.229 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.229 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.229 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.229 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.229 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.229 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_7weq2-_FEkT51PfhS1TopF2", 2020-11-09T18:15:53.229 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.230 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.230 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.230 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.230 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.231 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945595.466475_115.1", 2020-11-09T18:15:53.231 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.231 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-21", 2020-11-09T18:15:53.231 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:15.466475089Z", 2020-11-09T18:15:53.231 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.231 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.231 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.231 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.232 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.232 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_cxcDXnujWtafpwp1_Zf7_7E", 2020-11-09T18:15:53.232 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.232 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.232 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.233 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.233 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.233 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945595.470860_116.1", 2020-11-09T18:15:53.233 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.233 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-21", 2020-11-09T18:15:53.234 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:15.470860202Z", 2020-11-09T18:15:53.234 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.234 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.234 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.234 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.234 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.234 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_cxcDXnujWtafpwp1_Zf7_7E", 2020-11-09T18:15:53.235 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.235 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.235 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.235 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.235 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.236 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945595.574082_117.1", 2020-11-09T18:15:53.236 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.236 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-22:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.22", 2020-11-09T18:15:53.236 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:15.574082160Z", 2020-11-09T18:15:53.236 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.236 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.237 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.237 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.237 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.237 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_gu325NeoTQ-n6-N4RZO_Ufq", 2020-11-09T18:15:53.238 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.238 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.238 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.238 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.238 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.239 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945595.578263_118.1", 2020-11-09T18:15:53.239 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.239 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-22:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.22", 2020-11-09T18:15:53.239 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:15.578263069Z", 2020-11-09T18:15:53.239 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.239 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.240 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.240 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.240 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.240 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_gu325NeoTQ-n6-N4RZO_Ufq", 2020-11-09T18:15:53.240 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.240 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.241 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.241 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.241 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.241 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.241 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.241 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945595.580706_119.1", 2020-11-09T18:15:53.241 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.242 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-22", 2020-11-09T18:15:53.242 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:15.580706536Z", 2020-11-09T18:15:53.242 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.242 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.242 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.242 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.242 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.243 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_lHWYslSbF-coDhm_0XUYDqP", 2020-11-09T18:15:53.243 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.243 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.243 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.243 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.243 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.244 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945595.585376_120.1", 2020-11-09T18:15:53.244 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.244 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-22", 2020-11-09T18:15:53.244 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:15.585376092Z", 2020-11-09T18:15:53.244 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.245 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.245 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.245 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.245 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.245 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_lHWYslSbF-coDhm_0XUYDqP", 2020-11-09T18:15:53.245 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.246 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.246 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.246 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.246 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.246 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.246 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945595.651287_121.1", 2020-11-09T18:15:53.247 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.247 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-23:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.23", 2020-11-09T18:15:53.247 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:15.651287758Z", 2020-11-09T18:15:53.247 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.247 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.247 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.247 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.248 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.248 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_k8tGHuX9HMmJcr5VgC0V0M4", 2020-11-09T18:15:53.248 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.248 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.248 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.248 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.249 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.249 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.249 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945595.655554_122.1", 2020-11-09T18:15:53.249 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.249 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-23:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.23", 2020-11-09T18:15:53.249 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:15.655554732Z", 2020-11-09T18:15:53.250 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.250 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.250 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.250 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.250 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.250 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_k8tGHuX9HMmJcr5VgC0V0M4", 2020-11-09T18:15:53.251 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.251 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.251 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.251 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.251 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.252 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945595.657606_123.1", 2020-11-09T18:15:53.252 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.252 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-23", 2020-11-09T18:15:53.252 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:15.657606349Z", 2020-11-09T18:15:53.252 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.252 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.252 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.253 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.253 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.253 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_rDxg2fTzKxNRTnXPuogbkec", 2020-11-09T18:15:53.253 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.253 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.253 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.254 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.254 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.254 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.254 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.254 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945595.662266_124.1", 2020-11-09T18:15:53.254 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.254 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-23", 2020-11-09T18:15:53.255 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:15.662266951Z", 2020-11-09T18:15:53.255 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.255 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.255 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.255 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.255 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.255 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_rDxg2fTzKxNRTnXPuogbkec", 2020-11-09T18:15:53.256 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.256 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.256 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.256 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.256 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.257 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.257 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945595.694579_125.1", 2020-11-09T18:15:53.257 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.257 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-21:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.21", 2020-11-09T18:15:53.257 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:15.694579560Z", 2020-11-09T18:15:53.257 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.257 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.257 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_7weq2-_FEkT51PfhS1TopF2", 2020-11-09T18:15:53.258 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.258 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.258 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_7weq2-_FEkT51PfhS1TopF2", 2020-11-09T18:15:53.258 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.258 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.259 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.259 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.259 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.259 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.259 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.259 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945595.699216_126.1", 2020-11-09T18:15:53.259 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.259 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-21:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.21", 2020-11-09T18:15:53.260 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:15.699216478Z", 2020-11-09T18:15:53.260 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.260 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.260 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_7weq2-_FEkT51PfhS1TopF2", 2020-11-09T18:15:53.260 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.260 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.261 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_7weq2-_FEkT51PfhS1TopF2", 2020-11-09T18:15:53.261 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.261 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.261 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.261 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.261 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.262 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.262 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945595.704101_127.1", 2020-11-09T18:15:53.262 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.262 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-22:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.22", 2020-11-09T18:15:53.262 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:15.704101323Z", 2020-11-09T18:15:53.262 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.262 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.263 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_gu325NeoTQ-n6-N4RZO_Ufq", 2020-11-09T18:15:53.263 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.263 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.263 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_gu325NeoTQ-n6-N4RZO_Ufq", 2020-11-09T18:15:53.263 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.263 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.264 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.264 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.264 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.264 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.264 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945595.709327_128.1", 2020-11-09T18:15:53.264 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.265 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-22:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.22", 2020-11-09T18:15:53.265 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:15.709327694Z", 2020-11-09T18:15:53.265 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.265 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.265 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_gu325NeoTQ-n6-N4RZO_Ufq", 2020-11-09T18:15:53.265 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.265 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.266 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_gu325NeoTQ-n6-N4RZO_Ufq", 2020-11-09T18:15:53.266 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.266 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.266 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.266 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.266 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.267 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945595.722945_129.1", 2020-11-09T18:15:53.267 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.267 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-23:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.23", 2020-11-09T18:15:53.267 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:15.722945874Z", 2020-11-09T18:15:53.267 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.267 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.268 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_k8tGHuX9HMmJcr5VgC0V0M4", 2020-11-09T18:15:53.268 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.268 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.268 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_k8tGHuX9HMmJcr5VgC0V0M4", 2020-11-09T18:15:53.268 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.269 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.269 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.269 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.269 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.269 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945595.729371_130.1", 2020-11-09T18:15:53.269 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.270 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-23:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.23", 2020-11-09T18:15:53.270 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:15.729371675Z", 2020-11-09T18:15:53.270 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.270 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.270 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_k8tGHuX9HMmJcr5VgC0V0M4", 2020-11-09T18:15:53.270 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.271 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.271 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_k8tGHuX9HMmJcr5VgC0V0M4", 2020-11-09T18:15:53.271 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.271 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.271 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.271 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.272 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.272 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945597.242843_131.1", 2020-11-09T18:15:53.272 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.272 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-24:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.24", 2020-11-09T18:15:53.272 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:17.242843380Z", 2020-11-09T18:15:53.272 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.273 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.273 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.273 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.273 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.273 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_ZWaAaUyrIgJWu47a8QLknCB", 2020-11-09T18:15:53.273 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.274 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.274 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.274 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.274 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.274 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945597.247361_133.1", 2020-11-09T18:15:53.275 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.275 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-24:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.24", 2020-11-09T18:15:53.275 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:17.247361813Z", 2020-11-09T18:15:53.275 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.275 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.275 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.275 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.276 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.276 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_ZWaAaUyrIgJWu47a8QLknCB", 2020-11-09T18:15:53.276 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.276 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.276 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.277 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.277 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.277 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945597.249391_134.1", 2020-11-09T18:15:53.277 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.277 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-24", 2020-11-09T18:15:53.277 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:17.249391359Z", 2020-11-09T18:15:53.278 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.278 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.278 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.278 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.278 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.278 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_M6yzvodjSwrGlU0XXU23tmw", 2020-11-09T18:15:53.279 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.279 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.279 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.279 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.279 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.280 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945597.257906_135.1", 2020-11-09T18:15:53.280 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.280 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-24", 2020-11-09T18:15:53.280 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:17.257906751Z", 2020-11-09T18:15:53.280 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.280 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.280 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.281 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.281 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.281 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_M6yzvodjSwrGlU0XXU23tmw", 2020-11-09T18:15:53.281 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.281 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.282 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.282 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.282 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.282 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945598.021538_136.1", 2020-11-09T18:15:53.282 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.282 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-25:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.25", 2020-11-09T18:15:53.283 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:18.021538398Z", 2020-11-09T18:15:53.283 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.283 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.283 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.283 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.283 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.284 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dm5hw1Gq9w9JdDaVBV0h96_", 2020-11-09T18:15:53.284 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.284 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.284 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.284 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.284 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.285 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945598.026407_137.1", 2020-11-09T18:15:53.285 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.285 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-25:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.25", 2020-11-09T18:15:53.285 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:18.026407438Z", 2020-11-09T18:15:53.285 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.285 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.286 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.286 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.286 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.286 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dm5hw1Gq9w9JdDaVBV0h96_", 2020-11-09T18:15:53.286 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.286 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.287 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.287 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.287 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.287 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945598.028359_138.1", 2020-11-09T18:15:53.287 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.288 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-25", 2020-11-09T18:15:53.288 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:18.028359151Z", 2020-11-09T18:15:53.288 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.288 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.288 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.288 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.289 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.289 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_hJWxzyq2VXkrZe0lTr6UJ0X", 2020-11-09T18:15:53.289 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.289 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.289 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.289 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.289 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.290 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945598.032307_139.1", 2020-11-09T18:15:53.290 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.290 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-25", 2020-11-09T18:15:53.290 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:18.032307526Z", 2020-11-09T18:15:53.290 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.291 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.291 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.291 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.291 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.291 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_hJWxzyq2VXkrZe0lTr6UJ0X", 2020-11-09T18:15:53.291 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.292 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.292 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.292 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.292 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.292 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945598.347441_140.1", 2020-11-09T18:15:53.293 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.293 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-26:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.26", 2020-11-09T18:15:53.293 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:18.347441566Z", 2020-11-09T18:15:53.293 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.293 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.293 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.293 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.294 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.294 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_tUH5PREzBpkbjt0vmvKAgOc", 2020-11-09T18:15:53.294 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.294 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.294 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.294 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.295 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.295 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945598.351465_141.1", 2020-11-09T18:15:53.295 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.295 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-26:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.26", 2020-11-09T18:15:53.295 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:18.351465217Z", 2020-11-09T18:15:53.296 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.296 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.296 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.296 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.296 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.296 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_tUH5PREzBpkbjt0vmvKAgOc", 2020-11-09T18:15:53.296 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.297 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.297 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.297 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.297 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.298 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945598.353325_142.1", 2020-11-09T18:15:53.298 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.298 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-26", 2020-11-09T18:15:53.298 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:18.353325221Z", 2020-11-09T18:15:53.298 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.298 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.298 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.299 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.299 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.299 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_Z6tqFl2xPji_I9drTzIP2KZ", 2020-11-09T18:15:53.299 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.299 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.299 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.300 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.300 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.300 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945598.357392_143.1", 2020-11-09T18:15:53.300 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.300 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-26", 2020-11-09T18:15:53.300 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:18.357392493Z", 2020-11-09T18:15:53.301 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.301 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.301 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.301 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.301 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.301 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_Z6tqFl2xPji_I9drTzIP2KZ", 2020-11-09T18:15:53.302 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.302 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.302 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.302 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.302 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.303 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945598.450979_144.1", 2020-11-09T18:15:53.303 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.303 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-24:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.24", 2020-11-09T18:15:53.303 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:18.450979415Z", 2020-11-09T18:15:53.303 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.303 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.303 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_ZWaAaUyrIgJWu47a8QLknCB", 2020-11-09T18:15:53.304 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.304 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.304 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_ZWaAaUyrIgJWu47a8QLknCB", 2020-11-09T18:15:53.304 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.304 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.304 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.305 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.305 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.305 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945598.461497_145.1", 2020-11-09T18:15:53.305 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.305 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-24:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.24", 2020-11-09T18:15:53.306 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:18.461497100Z", 2020-11-09T18:15:53.306 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.306 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.306 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_ZWaAaUyrIgJWu47a8QLknCB", 2020-11-09T18:15:53.306 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.306 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.307 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_ZWaAaUyrIgJWu47a8QLknCB", 2020-11-09T18:15:53.307 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.307 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.307 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.307 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.307 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.308 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945598.470055_146.1", 2020-11-09T18:15:53.308 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.308 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-24:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.24", 2020-11-09T18:15:53.308 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:18.470055992Z", 2020-11-09T18:15:53.308 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.308 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.309 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_ZWaAaUyrIgJWu47a8QLknCB", 2020-11-09T18:15:53.309 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.309 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.309 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_ZWaAaUyrIgJWu47a8QLknCB", 2020-11-09T18:15:53.309 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 3 2020-11-09T18:15:53.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.309 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.310 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.310 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.310 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.310 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945598.478206_147.1", 2020-11-09T18:15:53.310 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.311 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-24:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.24", 2020-11-09T18:15:53.311 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:18.478206910Z", 2020-11-09T18:15:53.311 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.311 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.311 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_ZWaAaUyrIgJWu47a8QLknCB", 2020-11-09T18:15:53.311 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.312 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.312 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_ZWaAaUyrIgJWu47a8QLknCB", 2020-11-09T18:15:53.312 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 3 2020-11-09T18:15:53.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.312 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.312 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.312 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.312 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.313 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945598.482582_148.1", 2020-11-09T18:15:53.313 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.313 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-25:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.25", 2020-11-09T18:15:53.313 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:18.482582905Z", 2020-11-09T18:15:53.313 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.314 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.314 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dm5hw1Gq9w9JdDaVBV0h96_", 2020-11-09T18:15:53.314 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.314 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.314 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dm5hw1Gq9w9JdDaVBV0h96_", 2020-11-09T18:15:53.314 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.315 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.315 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.316 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.316 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.317 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945598.488285_149.1", 2020-11-09T18:15:53.317 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.317 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-25:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.25", 2020-11-09T18:15:53.318 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:18.488285331Z", 2020-11-09T18:15:53.318 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.318 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.318 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dm5hw1Gq9w9JdDaVBV0h96_", 2020-11-09T18:15:53.319 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.319 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.319 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dm5hw1Gq9w9JdDaVBV0h96_", 2020-11-09T18:15:53.319 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.320 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.320 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.320 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.321 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.322 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945598.495708_150.1", 2020-11-09T18:15:53.322 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.322 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-25:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.25", 2020-11-09T18:15:53.323 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:18.495708983Z", 2020-11-09T18:15:53.323 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.323 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.323 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dm5hw1Gq9w9JdDaVBV0h96_", 2020-11-09T18:15:53.324 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.324 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.324 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dm5hw1Gq9w9JdDaVBV0h96_", 2020-11-09T18:15:53.324 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 3 2020-11-09T18:15:53.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.325 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.325 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.325 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.325 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.326 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945598.502185_151.1", 2020-11-09T18:15:53.326 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.326 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-25:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.25", 2020-11-09T18:15:53.327 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:18.502185684Z", 2020-11-09T18:15:53.327 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.327 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.327 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dm5hw1Gq9w9JdDaVBV0h96_", 2020-11-09T18:15:53.328 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.328 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.328 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dm5hw1Gq9w9JdDaVBV0h96_", 2020-11-09T18:15:53.329 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 3 2020-11-09T18:15:53.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.329 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.329 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.330 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.330 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.331 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945598.514968_152.1", 2020-11-09T18:15:53.331 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.331 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-26:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.26", 2020-11-09T18:15:53.332 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:18.514968069Z", 2020-11-09T18:15:53.332 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.332 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.332 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_tUH5PREzBpkbjt0vmvKAgOc", 2020-11-09T18:15:53.333 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.333 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.333 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_tUH5PREzBpkbjt0vmvKAgOc", 2020-11-09T18:15:53.333 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.334 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.334 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.334 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.334 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.334 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.335 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945598.528310_153.1", 2020-11-09T18:15:53.335 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.335 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-26:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.26", 2020-11-09T18:15:53.335 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:18.528310207Z", 2020-11-09T18:15:53.335 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.335 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.335 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_tUH5PREzBpkbjt0vmvKAgOc", 2020-11-09T18:15:53.336 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.336 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.336 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_tUH5PREzBpkbjt0vmvKAgOc", 2020-11-09T18:15:53.336 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.336 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.336 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.337 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.337 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.337 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.337 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945598.543001_154.1", 2020-11-09T18:15:53.337 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.337 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-26:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.26", 2020-11-09T18:15:53.338 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:18.543001147Z", 2020-11-09T18:15:53.338 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.338 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.338 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_tUH5PREzBpkbjt0vmvKAgOc", 2020-11-09T18:15:53.338 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.338 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.339 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_tUH5PREzBpkbjt0vmvKAgOc", 2020-11-09T18:15:53.339 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 3 2020-11-09T18:15:53.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.339 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.339 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.339 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.339 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.340 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945598.548940_155.1", 2020-11-09T18:15:53.340 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.340 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-26:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.26", 2020-11-09T18:15:53.340 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:18.548940287Z", 2020-11-09T18:15:53.341 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.341 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.341 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_tUH5PREzBpkbjt0vmvKAgOc", 2020-11-09T18:15:53.341 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.341 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.342 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.342 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_tUH5PREzBpkbjt0vmvKAgOc", 2020-11-09T18:15:53.342 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 3 2020-11-09T18:15:53.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.343 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.343 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.343 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.343 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.344 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945600.502084_156.1", 2020-11-09T18:15:53.345 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.345 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-27:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.27", 2020-11-09T18:15:53.345 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:20.502084837Z", 2020-11-09T18:15:53.345 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.346 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.346 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.346 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.347 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.347 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_ffjKv4uq3V4IpykJfSlfwhB", 2020-11-09T18:15:53.347 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.347 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.348 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.348 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.348 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.349 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945600.507516_158.1", 2020-11-09T18:15:53.349 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.349 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-27:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.27", 2020-11-09T18:15:53.349 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:20.507516916Z", 2020-11-09T18:15:53.350 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.350 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.350 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.350 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.350 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.351 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_ffjKv4uq3V4IpykJfSlfwhB", 2020-11-09T18:15:53.351 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.352 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.352 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.352 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.352 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.353 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945600.509661_159.1", 2020-11-09T18:15:53.353 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.353 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-27", 2020-11-09T18:15:53.354 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:20.509661301Z", 2020-11-09T18:15:53.354 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.354 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.354 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.354 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.355 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.355 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_UJQ25D20xb-Xxms2LevItvh", 2020-11-09T18:15:53.355 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.355 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.355 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.355 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.356 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.356 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.356 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.356 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945600.513944_160.1", 2020-11-09T18:15:53.356 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.356 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-27", 2020-11-09T18:15:53.356 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:20.513944844Z", 2020-11-09T18:15:53.357 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.357 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.357 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.357 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.357 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.358 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_UJQ25D20xb-Xxms2LevItvh", 2020-11-09T18:15:53.358 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.358 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.358 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.358 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.358 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.359 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945601.434311_161.1", 2020-11-09T18:15:53.359 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.359 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-28:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.28", 2020-11-09T18:15:53.359 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:21.434311607Z", 2020-11-09T18:15:53.359 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.360 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.360 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.360 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.360 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.360 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_jFhIAKXmQdsa5R34lXgk3pv", 2020-11-09T18:15:53.360 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.361 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.361 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.361 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.361 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.361 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.361 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.361 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.361 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945601.438688_162.1", 2020-11-09T18:15:53.362 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.362 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-28:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.28", 2020-11-09T18:15:53.362 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:21.438688009Z", 2020-11-09T18:15:53.362 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.362 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.362 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.363 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.363 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.363 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_jFhIAKXmQdsa5R34lXgk3pv", 2020-11-09T18:15:53.363 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.364 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.364 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.364 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.364 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.364 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.365 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945601.440791_163.1", 2020-11-09T18:15:53.365 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.365 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-28", 2020-11-09T18:15:53.365 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:21.440791127Z", 2020-11-09T18:15:53.365 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.365 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.365 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.366 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.366 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.366 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.366 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_4xBDSjdegdCsfVupUBmzzRX", 2020-11-09T18:15:53.366 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.366 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.366 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.367 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.367 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.367 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.367 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945601.445125_164.1", 2020-11-09T18:15:53.367 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.368 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-28", 2020-11-09T18:15:53.368 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:21.445125336Z", 2020-11-09T18:15:53.368 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.368 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.368 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.368 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.369 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.369 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_4xBDSjdegdCsfVupUBmzzRX", 2020-11-09T18:15:53.369 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.369 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.369 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.369 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.370 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.370 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.370 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945603.023496_165.1", 2020-11-09T18:15:53.370 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.370 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-29:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.29", 2020-11-09T18:15:53.371 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:23.023496753Z", 2020-11-09T18:15:53.371 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.371 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.371 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.372 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.372 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.372 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_P83ahmV5nZ3tNcs0_lSHbDU", 2020-11-09T18:15:53.373 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.373 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.373 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.374 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.374 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.375 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945603.027781_166.1", 2020-11-09T18:15:53.375 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.375 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-29:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.29", 2020-11-09T18:15:53.375 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:23.027781495Z", 2020-11-09T18:15:53.376 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.376 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.376 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.376 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.377 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.377 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_P83ahmV5nZ3tNcs0_lSHbDU", 2020-11-09T18:15:53.377 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.378 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.378 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.379 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.379 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.380 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945603.030309_167.1", 2020-11-09T18:15:53.380 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.380 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-29", 2020-11-09T18:15:53.380 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:23.030309393Z", 2020-11-09T18:15:53.381 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.381 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.381 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.381 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.382 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.382 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_Hhh1A6pFRNzrIunHlFDZwos", 2020-11-09T18:15:53.383 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.383 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.383 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.384 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.384 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.385 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945603.034891_168.1", 2020-11-09T18:15:53.385 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.385 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-29", 2020-11-09T18:15:53.385 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:23.034891461Z", 2020-11-09T18:15:53.386 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.386 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.386 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.387 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.387 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.387 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_Hhh1A6pFRNzrIunHlFDZwos", 2020-11-09T18:15:53.388 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.388 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.389 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.389 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.389 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.390 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.390 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945603.213259_169.1", 2020-11-09T18:15:53.390 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.390 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-27:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.27", 2020-11-09T18:15:53.391 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:23.213259709Z", 2020-11-09T18:15:53.391 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.391 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.391 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_ffjKv4uq3V4IpykJfSlfwhB", 2020-11-09T18:15:53.392 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.392 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.393 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_ffjKv4uq3V4IpykJfSlfwhB", 2020-11-09T18:15:53.393 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.393 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.393 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.394 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.394 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.394 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.395 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.395 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945603.223295_170.1", 2020-11-09T18:15:53.395 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.396 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-27:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.27", 2020-11-09T18:15:53.396 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:23.223295591Z", 2020-11-09T18:15:53.396 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.396 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.397 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_ffjKv4uq3V4IpykJfSlfwhB", 2020-11-09T18:15:53.397 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.397 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.398 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_ffjKv4uq3V4IpykJfSlfwhB", 2020-11-09T18:15:53.398 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.398 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.399 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.399 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.399 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.399 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.401 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945604.018547_171.1", 2020-11-09T18:15:53.401 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.401 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-28:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.28", 2020-11-09T18:15:53.402 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:24.018547219Z", 2020-11-09T18:15:53.402 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.402 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.403 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_jFhIAKXmQdsa5R34lXgk3pv", 2020-11-09T18:15:53.403 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.404 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.404 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_jFhIAKXmQdsa5R34lXgk3pv", 2020-11-09T18:15:53.404 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.405 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.405 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.405 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.406 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.406 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.406 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945604.029428_172.1", 2020-11-09T18:15:53.407 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.407 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-28:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.28", 2020-11-09T18:15:53.407 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:24.029428921Z", 2020-11-09T18:15:53.408 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.408 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.408 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_jFhIAKXmQdsa5R34lXgk3pv", 2020-11-09T18:15:53.408 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.409 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.409 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_jFhIAKXmQdsa5R34lXgk3pv", 2020-11-09T18:15:53.410 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.410 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.410 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.411 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.411 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.412 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945604.720659_173.1", 2020-11-09T18:15:53.412 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.412 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-29:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.29", 2020-11-09T18:15:53.413 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:24.720659097Z", 2020-11-09T18:15:53.413 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.413 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.413 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_P83ahmV5nZ3tNcs0_lSHbDU", 2020-11-09T18:15:53.414 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.414 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.414 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.415 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_P83ahmV5nZ3tNcs0_lSHbDU", 2020-11-09T18:15:53.415 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.415 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.415 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.415 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.416 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.416 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.416 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945604.732717_174.1", 2020-11-09T18:15:53.416 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.416 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-29:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.29", 2020-11-09T18:15:53.416 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:24.732717989Z", 2020-11-09T18:15:53.417 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.417 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.417 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_P83ahmV5nZ3tNcs0_lSHbDU", 2020-11-09T18:15:53.418 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.418 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.418 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_P83ahmV5nZ3tNcs0_lSHbDU", 2020-11-09T18:15:53.418 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.419 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.419 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.419 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.419 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.419 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945605.406938_175.1", 2020-11-09T18:15:53.420 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.420 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-30:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.30", 2020-11-09T18:15:53.420 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:25.406938541Z", 2020-11-09T18:15:53.420 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.420 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.420 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.420 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.421 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.421 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_2k6PaixusxbzBO-jYw6D0Md", 2020-11-09T18:15:53.421 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.421 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.421 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.422 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.422 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.422 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.422 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945605.412027_176.1", 2020-11-09T18:15:53.422 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.422 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-30:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.30", 2020-11-09T18:15:53.422 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:25.412027440Z", 2020-11-09T18:15:53.423 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.423 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.423 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.423 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.423 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.423 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_2k6PaixusxbzBO-jYw6D0Md", 2020-11-09T18:15:53.424 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.424 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.424 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.424 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.424 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.425 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945605.413981_177.1", 2020-11-09T18:15:53.425 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.425 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-30", 2020-11-09T18:15:53.425 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:25.413981551Z", 2020-11-09T18:15:53.425 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.425 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.425 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.426 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.426 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.426 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_OVN11Cxzp_HRsHiBhSltQ2E", 2020-11-09T18:15:53.426 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.426 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.427 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.427 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.427 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.427 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945605.419171_178.1", 2020-11-09T18:15:53.427 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.428 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-30", 2020-11-09T18:15:53.428 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:25.419171989Z", 2020-11-09T18:15:53.428 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.428 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.428 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.428 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.428 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.429 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_OVN11Cxzp_HRsHiBhSltQ2E", 2020-11-09T18:15:53.429 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.429 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.429 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.429 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.429 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.430 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.430 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.430 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945605.813895_179.1", 2020-11-09T18:15:53.430 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.430 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-31:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.31", 2020-11-09T18:15:53.430 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:25.813895626Z", 2020-11-09T18:15:53.430 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.430 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.431 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.431 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.431 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.431 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dboGSnUFvBxtlhGOt6TYcuV", 2020-11-09T18:15:53.431 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.432 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.432 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.432 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.432 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.432 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945605.819203_180.1", 2020-11-09T18:15:53.433 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.433 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-31:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.31", 2020-11-09T18:15:53.433 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:25.819203440Z", 2020-11-09T18:15:53.433 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.433 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.433 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.433 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.434 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.434 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dboGSnUFvBxtlhGOt6TYcuV", 2020-11-09T18:15:53.434 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.434 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.434 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.434 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.435 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.435 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945605.821244_181.1", 2020-11-09T18:15:53.435 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.435 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-31", 2020-11-09T18:15:53.435 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:25.821244081Z", 2020-11-09T18:15:53.436 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.436 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.436 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.436 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.436 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.436 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_D_zBqUuf_PUMl6KPscj9tcq", 2020-11-09T18:15:53.437 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.437 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.437 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.437 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.437 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.438 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945605.825854_182.1", 2020-11-09T18:15:53.438 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.438 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-31", 2020-11-09T18:15:53.438 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:25.825854764Z", 2020-11-09T18:15:53.438 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.438 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.438 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.439 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.439 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.439 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_D_zBqUuf_PUMl6KPscj9tcq", 2020-11-09T18:15:53.439 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.439 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.440 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.440 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.440 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.440 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.314889_183.1", 2020-11-09T18:15:53.440 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.440 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-32:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.32", 2020-11-09T18:15:53.441 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.314889240Z", 2020-11-09T18:15:53.441 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.441 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.441 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.441 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.441 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.442 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_h7iYndUz6ZZeJDRnCAJhq6b", 2020-11-09T18:15:53.442 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.442 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.442 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.442 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.442 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.443 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.319791_184.1", 2020-11-09T18:15:53.443 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.443 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-32:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.32", 2020-11-09T18:15:53.443 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.319791421Z", 2020-11-09T18:15:53.443 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.443 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.444 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.444 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.444 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.444 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_h7iYndUz6ZZeJDRnCAJhq6b", 2020-11-09T18:15:53.444 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.445 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.445 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.445 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.445 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.445 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.321799_185.1", 2020-11-09T18:15:53.446 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.446 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-32", 2020-11-09T18:15:53.446 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.321799710Z", 2020-11-09T18:15:53.446 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.446 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.446 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.446 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.447 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.447 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_kfgLDq_uLGT_Ii4AzNlTUJd", 2020-11-09T18:15:53.447 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.447 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.447 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.447 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.448 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.448 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.326129_186.1", 2020-11-09T18:15:53.448 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.448 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-32", 2020-11-09T18:15:53.448 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.326129812Z", 2020-11-09T18:15:53.449 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.449 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.449 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.449 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.449 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.449 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_kfgLDq_uLGT_Ii4AzNlTUJd", 2020-11-09T18:15:53.449 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.450 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.450 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.450 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.450 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.450 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.451 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.351846_187.1", 2020-11-09T18:15:53.451 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.451 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-30:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.30", 2020-11-09T18:15:53.451 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.351846735Z", 2020-11-09T18:15:53.451 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.451 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.451 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_2k6PaixusxbzBO-jYw6D0Md", 2020-11-09T18:15:53.451 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.452 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.452 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_2k6PaixusxbzBO-jYw6D0Md", 2020-11-09T18:15:53.452 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.452 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.452 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.453 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.453 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.453 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.358962_188.1", 2020-11-09T18:15:53.453 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.453 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-30:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.30", 2020-11-09T18:15:53.453 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.358962403Z", 2020-11-09T18:15:53.454 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.454 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.454 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_2k6PaixusxbzBO-jYw6D0Md", 2020-11-09T18:15:53.454 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.454 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.454 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_2k6PaixusxbzBO-jYw6D0Md", 2020-11-09T18:15:53.455 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.455 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.455 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.455 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.455 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.456 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.413199_189.1", 2020-11-09T18:15:53.456 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.456 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-31:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.31", 2020-11-09T18:15:53.456 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.413199050Z", 2020-11-09T18:15:53.456 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.456 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.457 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dboGSnUFvBxtlhGOt6TYcuV", 2020-11-09T18:15:53.457 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.457 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.457 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dboGSnUFvBxtlhGOt6TYcuV", 2020-11-09T18:15:53.457 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.458 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.458 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.458 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.458 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.458 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.423298_190.1", 2020-11-09T18:15:53.458 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.459 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-31:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.31", 2020-11-09T18:15:53.459 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.423298203Z", 2020-11-09T18:15:53.459 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.459 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.459 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dboGSnUFvBxtlhGOt6TYcuV", 2020-11-09T18:15:53.459 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.460 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.460 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dboGSnUFvBxtlhGOt6TYcuV", 2020-11-09T18:15:53.460 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.460 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.460 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.460 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.460 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.461 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.461 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.486367_191.1", 2020-11-09T18:15:53.461 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.461 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-32:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.32", 2020-11-09T18:15:53.461 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.486367158Z", 2020-11-09T18:15:53.461 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.462 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.462 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_h7iYndUz6ZZeJDRnCAJhq6b", 2020-11-09T18:15:53.462 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.462 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.462 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_h7iYndUz6ZZeJDRnCAJhq6b", 2020-11-09T18:15:53.462 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.463 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.463 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.463 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.463 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.463 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.463 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.497483_192.1", 2020-11-09T18:15:53.464 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.464 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-32:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.32", 2020-11-09T18:15:53.464 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.497483092Z", 2020-11-09T18:15:53.464 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.464 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.464 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_h7iYndUz6ZZeJDRnCAJhq6b", 2020-11-09T18:15:53.464 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.465 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.465 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_h7iYndUz6ZZeJDRnCAJhq6b", 2020-11-09T18:15:53.465 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.465 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.465 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.466 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.466 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.466 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.466 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.466 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.567936_193.1", 2020-11-09T18:15:53.466 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.466 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-33:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.33", 2020-11-09T18:15:53.467 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.567936942Z", 2020-11-09T18:15:53.467 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.467 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.467 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.467 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.467 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.467 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dV7Zwb-EMDhOHgweLJlLD79", 2020-11-09T18:15:53.468 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.468 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.468 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.468 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.468 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.469 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.469 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.573179_194.1", 2020-11-09T18:15:53.469 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.469 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-33:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.33", 2020-11-09T18:15:53.469 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.573179777Z", 2020-11-09T18:15:53.469 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.469 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.469 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.470 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.470 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.470 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dV7Zwb-EMDhOHgweLJlLD79", 2020-11-09T18:15:53.470 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.470 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.471 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.471 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.471 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.471 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.471 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.575137_195.1", 2020-11-09T18:15:53.471 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.472 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-33", 2020-11-09T18:15:53.472 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.575137143Z", 2020-11-09T18:15:53.472 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.472 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.472 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.472 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.472 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.472 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.473 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "__9pytw2itoqGRYgrLwyaE0v", 2020-11-09T18:15:53.473 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.473 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.473 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.473 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.473 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.474 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.474 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.578921_196.1", 2020-11-09T18:15:53.474 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.474 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-33", 2020-11-09T18:15:53.474 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.578921064Z", 2020-11-09T18:15:53.474 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.475 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.475 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.475 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.475 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.475 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "__9pytw2itoqGRYgrLwyaE0v", 2020-11-09T18:15:53.475 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.476 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.476 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.476 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.476 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.476 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.477 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.603950_198.1", 2020-11-09T18:15:53.477 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.477 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-34:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.34", 2020-11-09T18:15:53.477 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.603950696Z", 2020-11-09T18:15:53.477 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.477 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.477 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.477 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.478 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.478 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_KazzTgKF9mHRl9t-mjqUZIF", 2020-11-09T18:15:53.478 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.478 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.478 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.479 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.479 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.479 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.479 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.608737_199.1", 2020-11-09T18:15:53.479 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.479 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-34:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.34", 2020-11-09T18:15:53.480 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.608737440Z", 2020-11-09T18:15:53.480 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.480 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.480 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.480 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.480 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.480 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_KazzTgKF9mHRl9t-mjqUZIF", 2020-11-09T18:15:53.481 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.481 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.481 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.481 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.481 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.481 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.482 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.610541_200.1", 2020-11-09T18:15:53.482 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.482 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-34", 2020-11-09T18:15:53.482 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.610541395Z", 2020-11-09T18:15:53.482 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.482 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.482 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.483 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.483 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.483 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.483 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_QkHzIdBeCUUuKSc9kaJ3NT5", 2020-11-09T18:15:53.483 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.483 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.483 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.484 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.484 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.484 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.484 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.484 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.484 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.614352_201.1", 2020-11-09T18:15:53.484 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.484 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-34", 2020-11-09T18:15:53.485 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.614352129Z", 2020-11-09T18:15:53.485 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.485 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.485 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.485 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.485 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.486 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_QkHzIdBeCUUuKSc9kaJ3NT5", 2020-11-09T18:15:53.486 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.486 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.486 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.486 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.486 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.487 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.658773_202.1", 2020-11-09T18:15:53.487 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.487 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-35:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.35", 2020-11-09T18:15:53.487 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.658773780Z", 2020-11-09T18:15:53.487 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.488 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.488 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.488 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.488 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.488 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dAiojqzewqe1JboY85aFSyJ", 2020-11-09T18:15:53.488 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.489 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.489 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.489 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.489 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.489 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.489 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.663192_203.1", 2020-11-09T18:15:53.490 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.490 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-35:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.35", 2020-11-09T18:15:53.490 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.663192679Z", 2020-11-09T18:15:53.490 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.490 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.490 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.490 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.491 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.491 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dAiojqzewqe1JboY85aFSyJ", 2020-11-09T18:15:53.491 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.492 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.492 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.492 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.492 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.492 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.493 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.665284_204.1", 2020-11-09T18:15:53.493 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.493 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-35", 2020-11-09T18:15:53.493 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.665284756Z", 2020-11-09T18:15:53.493 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.493 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.494 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.494 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.494 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.494 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.494 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_YIG4M8hwsEikvpJIsO0cCiX", 2020-11-09T18:15:53.494 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.495 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.495 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.495 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.495 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.495 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.671390_205.1", 2020-11-09T18:15:53.496 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.496 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-35", 2020-11-09T18:15:53.496 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.671390803Z", 2020-11-09T18:15:53.496 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.496 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.496 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.496 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.497 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.497 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_YIG4M8hwsEikvpJIsO0cCiX", 2020-11-09T18:15:53.497 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.497 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.498 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.498 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.498 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.498 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.498 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.498 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.708610_206.1", 2020-11-09T18:15:53.498 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.499 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-33:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.33", 2020-11-09T18:15:53.499 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.708610357Z", 2020-11-09T18:15:53.499 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.499 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.499 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dV7Zwb-EMDhOHgweLJlLD79", 2020-11-09T18:15:53.499 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.500 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.500 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dV7Zwb-EMDhOHgweLJlLD79", 2020-11-09T18:15:53.500 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.500 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.500 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.500 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.501 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.501 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945606.713328_207.1", 2020-11-09T18:15:53.501 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.502 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-33:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.33", 2020-11-09T18:15:53.502 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:26.713328296Z", 2020-11-09T18:15:53.502 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.502 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.502 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dV7Zwb-EMDhOHgweLJlLD79", 2020-11-09T18:15:53.502 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.503 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.503 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dV7Zwb-EMDhOHgweLJlLD79", 2020-11-09T18:15:53.503 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.503 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.504 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.504 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.504 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.504 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945607.760169_210.1", 2020-11-09T18:15:53.504 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.505 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-34:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.34", 2020-11-09T18:15:53.505 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:27.760169945Z", 2020-11-09T18:15:53.505 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.505 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.505 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_KazzTgKF9mHRl9t-mjqUZIF", 2020-11-09T18:15:53.505 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.506 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.506 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_KazzTgKF9mHRl9t-mjqUZIF", 2020-11-09T18:15:53.506 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.506 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.506 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.507 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.507 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.507 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945607.771847_211.1", 2020-11-09T18:15:53.507 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.508 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-34:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.34", 2020-11-09T18:15:53.508 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:27.771847568Z", 2020-11-09T18:15:53.508 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.508 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.508 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_KazzTgKF9mHRl9t-mjqUZIF", 2020-11-09T18:15:53.508 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.509 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.509 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_KazzTgKF9mHRl9t-mjqUZIF", 2020-11-09T18:15:53.509 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.509 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.509 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.509 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.510 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.510 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.510 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945609.277502_212.1", 2020-11-09T18:15:53.510 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.510 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-35:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.35", 2020-11-09T18:15:53.511 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:29.277502241Z", 2020-11-09T18:15:53.511 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.511 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.511 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dAiojqzewqe1JboY85aFSyJ", 2020-11-09T18:15:53.511 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.511 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.512 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dAiojqzewqe1JboY85aFSyJ", 2020-11-09T18:15:53.512 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.512 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.512 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.512 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.512 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.513 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.513 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945609.288552_213.1", 2020-11-09T18:15:53.513 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.513 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-35:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.35", 2020-11-09T18:15:53.513 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:29.288552016Z", 2020-11-09T18:15:53.513 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.514 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.514 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dAiojqzewqe1JboY85aFSyJ", 2020-11-09T18:15:53.514 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.514 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.514 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dAiojqzewqe1JboY85aFSyJ", 2020-11-09T18:15:53.515 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.515 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.515 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.515 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.515 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.516 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945609.660249_214.1", 2020-11-09T18:15:53.516 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.516 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-36:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.36", 2020-11-09T18:15:53.516 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:29.660249505Z", 2020-11-09T18:15:53.516 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.517 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.517 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.517 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.517 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.517 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_ozJuqjR3GVh8aGnB2Xh4hR5", 2020-11-09T18:15:53.518 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.518 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.518 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.518 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.518 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.519 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945609.665942_215.1", 2020-11-09T18:15:53.519 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.519 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-36:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.36", 2020-11-09T18:15:53.519 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:29.665942367Z", 2020-11-09T18:15:53.519 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.519 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.520 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.520 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.520 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.520 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_ozJuqjR3GVh8aGnB2Xh4hR5", 2020-11-09T18:15:53.520 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.521 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.521 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.521 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.521 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.521 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945609.669229_216.1", 2020-11-09T18:15:53.522 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.522 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-36", 2020-11-09T18:15:53.522 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:29.669229416Z", 2020-11-09T18:15:53.522 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.522 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.522 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.523 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.523 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.523 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_DZ1KbRgq_ERertkZGzu4RA_", 2020-11-09T18:15:53.523 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.523 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.523 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.524 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.524 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.524 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945609.675301_217.1", 2020-11-09T18:15:53.524 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.524 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-36", 2020-11-09T18:15:53.525 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:29.675301279Z", 2020-11-09T18:15:53.525 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.525 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.525 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.525 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.525 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.526 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_DZ1KbRgq_ERertkZGzu4RA_", 2020-11-09T18:15:53.526 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.526 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.526 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.526 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.526 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.527 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945610.079342_218.1", 2020-11-09T18:15:53.527 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.527 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-37:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.37", 2020-11-09T18:15:53.527 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:30.079342479Z", 2020-11-09T18:15:53.527 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.528 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.528 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.528 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.528 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.528 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_fx1DINtpIECPzOGaSfs3ucL", 2020-11-09T18:15:53.529 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.529 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.529 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.529 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.529 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.530 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.530 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945610.083364_219.1", 2020-11-09T18:15:53.530 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.530 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-37:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.37", 2020-11-09T18:15:53.530 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:30.083364883Z", 2020-11-09T18:15:53.530 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.531 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.531 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.531 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.531 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.531 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_fx1DINtpIECPzOGaSfs3ucL", 2020-11-09T18:15:53.531 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.532 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.532 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.532 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.532 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.533 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945610.085275_220.1", 2020-11-09T18:15:53.533 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.533 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-37", 2020-11-09T18:15:53.533 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:30.085275759Z", 2020-11-09T18:15:53.533 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.533 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.533 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.534 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.534 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.534 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_J2nfKVSwwp-9vo2WCGxBAhy", 2020-11-09T18:15:53.534 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.534 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.535 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.535 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.535 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.535 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945610.091583_221.1", 2020-11-09T18:15:53.535 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.536 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-37", 2020-11-09T18:15:53.536 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:30.091583370Z", 2020-11-09T18:15:53.536 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.536 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.536 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.536 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.536 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.537 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_J2nfKVSwwp-9vo2WCGxBAhy", 2020-11-09T18:15:53.537 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.537 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.537 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.537 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.538 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.538 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945610.137385_222.1", 2020-11-09T18:15:53.538 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.538 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-38:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.38", 2020-11-09T18:15:53.539 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:30.137385505Z", 2020-11-09T18:15:53.539 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.539 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.539 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.539 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.540 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.540 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_MKdJSqznasfgUfD8pSqNyMN", 2020-11-09T18:15:53.540 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.540 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.540 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.541 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.541 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.541 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945610.141970_223.1", 2020-11-09T18:15:53.541 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.541 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-38:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.38", 2020-11-09T18:15:53.542 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:30.141970597Z", 2020-11-09T18:15:53.542 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.542 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.542 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.542 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.543 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.543 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_MKdJSqznasfgUfD8pSqNyMN", 2020-11-09T18:15:53.543 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.543 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.543 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.544 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.544 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.544 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945610.144161_224.1", 2020-11-09T18:15:53.544 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.545 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-38", 2020-11-09T18:15:53.545 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:30.144161714Z", 2020-11-09T18:15:53.545 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.545 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.545 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.545 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.546 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.546 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_Lsx6lFhkZQ0sbnB7SgpHP7X", 2020-11-09T18:15:53.546 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.546 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.546 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.547 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.547 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.547 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945610.148613_225.1", 2020-11-09T18:15:53.547 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.548 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-38", 2020-11-09T18:15:53.548 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:30.148613440Z", 2020-11-09T18:15:53.548 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.548 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.548 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.548 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.549 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.549 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_Lsx6lFhkZQ0sbnB7SgpHP7X", 2020-11-09T18:15:53.549 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.549 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.549 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.549 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.550 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.550 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945611.763559_226.1", 2020-11-09T18:15:53.550 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.550 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-39:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.39", 2020-11-09T18:15:53.551 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:31.763559546Z", 2020-11-09T18:15:53.551 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.551 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.551 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.551 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.552 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.552 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_DoCr53MUjfmeA1PTWsLAoZg", 2020-11-09T18:15:53.552 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.552 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.552 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.553 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.553 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.553 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945611.768768_228.1", 2020-11-09T18:15:53.553 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.553 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-39:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.39", 2020-11-09T18:15:53.554 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:31.768768717Z", 2020-11-09T18:15:53.554 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.554 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.554 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.554 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.554 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.555 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_DoCr53MUjfmeA1PTWsLAoZg", 2020-11-09T18:15:53.555 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.555 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.555 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.555 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.555 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.556 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945611.771007_229.1", 2020-11-09T18:15:53.556 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.556 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-39", 2020-11-09T18:15:53.556 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:31.771007696Z", 2020-11-09T18:15:53.556 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.557 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.557 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.557 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.557 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.557 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_pGgwgGXPCFeV9dl7j-mkG2D", 2020-11-09T18:15:53.557 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.558 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.558 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.558 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.558 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.559 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945611.776767_230.1", 2020-11-09T18:15:53.559 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.559 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-39", 2020-11-09T18:15:53.559 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:31.776767187Z", 2020-11-09T18:15:53.559 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.559 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.560 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.560 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.560 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.560 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_pGgwgGXPCFeV9dl7j-mkG2D", 2020-11-09T18:15:53.560 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.561 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.561 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.561 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.561 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.562 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945612.536576_231.1", 2020-11-09T18:15:53.562 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.562 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-40:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.40", 2020-11-09T18:15:53.562 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:32.536576361Z", 2020-11-09T18:15:53.562 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.562 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.563 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.563 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.563 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.563 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_eRiB6BdOLC_U7ajfFCuPyzq", 2020-11-09T18:15:53.563 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.564 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.564 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.564 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.564 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.565 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945612.542749_232.1", 2020-11-09T18:15:53.565 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.565 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-40:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.40", 2020-11-09T18:15:53.565 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:32.542749842Z", 2020-11-09T18:15:53.565 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.565 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.566 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.566 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.566 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.566 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_eRiB6BdOLC_U7ajfFCuPyzq", 2020-11-09T18:15:53.566 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.567 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.567 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.567 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.567 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.568 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945612.545398_233.1", 2020-11-09T18:15:53.568 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.568 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-40", 2020-11-09T18:15:53.568 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:32.545398453Z", 2020-11-09T18:15:53.568 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.568 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.569 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.569 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.569 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.569 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_v-PHMVkQGnPvIMVgQXzEv-e", 2020-11-09T18:15:53.569 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.570 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.570 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.570 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.570 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.571 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945612.550287_234.1", 2020-11-09T18:15:53.571 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.571 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-40", 2020-11-09T18:15:53.571 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:32.550287369Z", 2020-11-09T18:15:53.571 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.571 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.572 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.572 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.572 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.572 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_v-PHMVkQGnPvIMVgQXzEv-e", 2020-11-09T18:15:53.572 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.573 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.573 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.573 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.573 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.573 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945612.641557_236.1", 2020-11-09T18:15:53.574 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.575 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-41:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.41", 2020-11-09T18:15:53.575 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:32.641557244Z", 2020-11-09T18:15:53.575 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.575 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.575 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.576 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.576 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.576 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_VbEzzl6PqzRKUgwPXZGxMNK", 2020-11-09T18:15:53.576 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.576 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.577 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.577 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.577 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.577 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945612.646933_237.1", 2020-11-09T18:15:53.578 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.578 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-41:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.41", 2020-11-09T18:15:53.578 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:32.646933508Z", 2020-11-09T18:15:53.578 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.578 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.578 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.579 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.579 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.579 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_VbEzzl6PqzRKUgwPXZGxMNK", 2020-11-09T18:15:53.579 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.580 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.580 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.580 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.580 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.581 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945612.649001_238.1", 2020-11-09T18:15:53.581 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.581 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-41", 2020-11-09T18:15:53.581 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:32.649001607Z", 2020-11-09T18:15:53.581 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.581 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.581 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.582 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.582 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.582 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_-QlJbrN7eIOTl-coxNoWB0_", 2020-11-09T18:15:53.582 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.583 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.583 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.583 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.583 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.584 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945612.653005_239.1", 2020-11-09T18:15:53.584 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.584 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-41", 2020-11-09T18:15:53.584 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:32.653005302Z", 2020-11-09T18:15:53.584 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.584 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.585 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.585 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.585 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.585 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_-QlJbrN7eIOTl-coxNoWB0_", 2020-11-09T18:15:53.585 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.586 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.586 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.586 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.586 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.586 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945563.557787_47.1", 2020-11-09T18:15:53.587 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.587 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-1:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.1", 2020-11-09T18:15:53.587 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:43.557787352Z", 2020-11-09T18:15:53.587 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.587 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.587 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.587 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.588 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.588 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_5-8ThY0rD5nJEgi4q-i6qW8", 2020-11-09T18:15:53.588 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.588 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.588 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.589 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.589 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.589 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945563.567624_51.1", 2020-11-09T18:15:53.589 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.589 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-1:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.1", 2020-11-09T18:15:53.590 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:43.567624280Z", 2020-11-09T18:15:53.590 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.590 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.590 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.590 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.590 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.591 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_5-8ThY0rD5nJEgi4q-i6qW8", 2020-11-09T18:15:53.591 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.591 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.591 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.591 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.592 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.592 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945563.569310_52.1", 2020-11-09T18:15:53.592 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.592 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-1", 2020-11-09T18:15:53.593 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:43.569310797Z", 2020-11-09T18:15:53.593 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.593 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.593 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.593 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.594 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.594 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_KrRBCosiXlBz0ajusgkrpWS", 2020-11-09T18:15:53.594 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.594 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.594 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.594 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.595 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.595 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945563.577890_53.1", 2020-11-09T18:15:53.595 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.595 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-1", 2020-11-09T18:15:53.596 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:43.577890725Z", 2020-11-09T18:15:53.596 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.596 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.596 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.596 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.597 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.597 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_KrRBCosiXlBz0ajusgkrpWS", 2020-11-09T18:15:53.597 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.597 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.597 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.597 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.598 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.599 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945563.608973_54.1", 2020-11-09T18:15:53.599 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.599 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-2:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.2", 2020-11-09T18:15:53.599 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:43.608973238Z", 2020-11-09T18:15:53.599 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.599 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.600 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.600 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.600 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.600 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_eZ3F6MzV7ES6qOIMaFzRrOO", 2020-11-09T18:15:53.600 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.601 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.601 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.601 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.601 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.601 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945563.618816_55.1", 2020-11-09T18:15:53.602 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.602 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-2:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.2", 2020-11-09T18:15:53.602 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:43.618816140Z", 2020-11-09T18:15:53.602 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.602 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.602 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.603 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.603 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.603 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_eZ3F6MzV7ES6qOIMaFzRrOO", 2020-11-09T18:15:53.603 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.603 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.604 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.604 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.604 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.604 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.604 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945563.621200_56.1", 2020-11-09T18:15:53.604 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.605 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-2", 2020-11-09T18:15:53.605 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:43.621200541Z", 2020-11-09T18:15:53.605 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.605 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.605 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.605 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.606 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.606 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_rj9KSIrvtNvXYWU2UYD1Z_m", 2020-11-09T18:15:53.606 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.606 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.606 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.606 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.607 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.607 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.607 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945563.627302_57.1", 2020-11-09T18:15:53.607 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.607 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-2", 2020-11-09T18:15:53.607 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:43.627302782Z", 2020-11-09T18:15:53.608 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.608 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.608 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.608 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.608 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.608 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_rj9KSIrvtNvXYWU2UYD1Z_m", 2020-11-09T18:15:53.609 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.609 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.609 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.609 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.609 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.610 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945567.627267_58.1", 2020-11-09T18:15:53.610 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.610 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-3:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.3", 2020-11-09T18:15:53.610 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:47.627267860Z", 2020-11-09T18:15:53.610 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.611 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.611 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.611 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.611 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.611 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_TnwqP2_mu3KZnFJOXhnO-DD", 2020-11-09T18:15:53.611 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.612 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.612 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.612 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.612 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.613 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945567.630834_59.1", 2020-11-09T18:15:53.613 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.613 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-3:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.3", 2020-11-09T18:15:53.613 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:47.630834733Z", 2020-11-09T18:15:53.613 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.613 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.614 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.614 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.614 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.614 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_TnwqP2_mu3KZnFJOXhnO-DD", 2020-11-09T18:15:53.614 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.615 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.615 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.615 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.615 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.615 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.616 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945567.632589_60.1", 2020-11-09T18:15:53.616 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.616 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-3", 2020-11-09T18:15:53.616 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:47.632589277Z", 2020-11-09T18:15:53.616 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.616 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.616 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.617 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.617 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.617 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_MHzcuL8U9yY5bsA8iZvWhTc", 2020-11-09T18:15:53.617 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.617 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.618 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.618 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.618 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.618 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945567.636634_61.1", 2020-11-09T18:15:53.618 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.619 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-3", 2020-11-09T18:15:53.619 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:47.636634661Z", 2020-11-09T18:15:53.619 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.619 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.619 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.619 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.620 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.620 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_MHzcuL8U9yY5bsA8iZvWhTc", 2020-11-09T18:15:53.620 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.620 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.620 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.620 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.621 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.621 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945570.370676_62.1", 2020-11-09T18:15:53.621 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.621 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-4:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.4", 2020-11-09T18:15:53.621 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:50.370676662Z", 2020-11-09T18:15:53.622 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.622 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.622 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.622 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.622 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.623 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_ufcC505t7ROkSzsmsL1irho", 2020-11-09T18:15:53.623 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.623 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.623 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.623 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.623 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.624 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.624 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945570.375714_64.1", 2020-11-09T18:15:53.624 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.624 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-4:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.4", 2020-11-09T18:15:53.624 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:50.375714519Z", 2020-11-09T18:15:53.624 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.625 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.625 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.625 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.625 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.625 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_ufcC505t7ROkSzsmsL1irho", 2020-11-09T18:15:53.625 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.626 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.626 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.626 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.626 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.626 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.627 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945570.377850_65.1", 2020-11-09T18:15:53.627 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.627 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-4", 2020-11-09T18:15:53.627 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:50.377850670Z", 2020-11-09T18:15:53.627 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.627 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.627 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.628 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.628 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.628 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_xnR_YIJ8zUxxn61yz_DTUkk", 2020-11-09T18:15:53.628 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.628 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.629 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.629 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.629 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.629 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945570.382459_66.1", 2020-11-09T18:15:53.629 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.630 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-4", 2020-11-09T18:15:53.630 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:50.382459732Z", 2020-11-09T18:15:53.630 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.630 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.630 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.630 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.631 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.631 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_xnR_YIJ8zUxxn61yz_DTUkk", 2020-11-09T18:15:53.631 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.631 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.631 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.632 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.632 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.632 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.632 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945570.406070_67.1", 2020-11-09T18:15:53.632 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.632 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-5:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.5", 2020-11-09T18:15:53.633 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:50.406070925Z", 2020-11-09T18:15:53.633 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.633 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.633 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.633 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.633 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.634 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_UShZGGBB4kBaxRzXtVmZBUW", 2020-11-09T18:15:53.634 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.634 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.634 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.634 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.634 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.635 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.635 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945570.410945_68.1", 2020-11-09T18:15:53.635 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.635 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-5:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.5", 2020-11-09T18:15:53.635 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:50.410945255Z", 2020-11-09T18:15:53.636 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.636 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.636 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.636 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.636 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.636 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_UShZGGBB4kBaxRzXtVmZBUW", 2020-11-09T18:15:53.637 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.637 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.637 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.637 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.637 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.638 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945570.412796_69.1", 2020-11-09T18:15:53.638 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.638 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-5", 2020-11-09T18:15:53.638 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:50.412796913Z", 2020-11-09T18:15:53.638 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.638 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.639 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.639 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.639 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.639 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_5bx-BH8ijLijK0Hb5tRXntZ", 2020-11-09T18:15:53.639 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.640 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.640 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.640 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.640 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.640 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945570.416832_70.1", 2020-11-09T18:15:53.641 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.641 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-5", 2020-11-09T18:15:53.641 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:50.416832729Z", 2020-11-09T18:15:53.641 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.641 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.641 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.641 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.642 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.642 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_5bx-BH8ijLijK0Hb5tRXntZ", 2020-11-09T18:15:53.642 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.642 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.642 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.643 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.643 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.643 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945570.461105_71.1", 2020-11-09T18:15:53.643 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.644 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-6:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.6", 2020-11-09T18:15:53.644 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:50.461105755Z", 2020-11-09T18:15:53.644 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.644 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.644 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.644 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.645 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.645 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.645 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_qWlHsdsWYJHc0mQkDC3vzYx", 2020-11-09T18:15:53.645 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.645 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.645 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.645 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.646 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.646 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.646 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.646 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945570.473537_72.1", 2020-11-09T18:15:53.646 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.646 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-6:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.6", 2020-11-09T18:15:53.647 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:50.473537938Z", 2020-11-09T18:15:53.647 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.647 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.647 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.647 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.647 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.648 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_qWlHsdsWYJHc0mQkDC3vzYx", 2020-11-09T18:15:53.648 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.648 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.648 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.648 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.648 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.649 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945570.477860_73.1", 2020-11-09T18:15:53.649 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.649 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-6", 2020-11-09T18:15:53.649 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:50.477860682Z", 2020-11-09T18:15:53.649 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.650 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.650 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.650 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.650 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.650 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dmuyyeGa1O--zFbpSa0SA8D", 2020-11-09T18:15:53.650 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.651 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.651 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.651 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.651 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.652 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945570.489122_74.1", 2020-11-09T18:15:53.652 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.652 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-6", 2020-11-09T18:15:53.652 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:50.489122994Z", 2020-11-09T18:15:53.652 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.652 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.652 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.653 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.653 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.653 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_dmuyyeGa1O--zFbpSa0SA8D", 2020-11-09T18:15:53.653 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.653 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.654 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.654 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.654 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.654 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945579.353942_75.1", 2020-11-09T18:15:53.654 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.655 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-7:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.7", 2020-11-09T18:15:53.655 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:59.353942489Z", 2020-11-09T18:15:53.655 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.655 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.655 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.655 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.656 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.656 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_GIkCDw3GWmDRO3-A0auRtxB", 2020-11-09T18:15:53.656 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.656 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.656 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.656 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.657 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.657 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945579.359041_78.1", 2020-11-09T18:15:53.657 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.657 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-7:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.7", 2020-11-09T18:15:53.657 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:59.359041450Z", 2020-11-09T18:15:53.658 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.658 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.658 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.658 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.658 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.658 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_GIkCDw3GWmDRO3-A0auRtxB", 2020-11-09T18:15:53.659 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.659 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.659 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.659 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.659 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.660 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945579.361526_79.1", 2020-11-09T18:15:53.660 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.660 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-7", 2020-11-09T18:15:53.660 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:59.361526900Z", 2020-11-09T18:15:53.660 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.660 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.661 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.661 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.661 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.661 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_euurhSSx6WmV693QUVvezHc", 2020-11-09T18:15:53.661 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.662 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.662 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.662 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.662 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.663 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945579.367322_80.1", 2020-11-09T18:15:53.663 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.663 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-7", 2020-11-09T18:15:53.663 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:59.367322143Z", 2020-11-09T18:15:53.663 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.663 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.663 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.664 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.664 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.664 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_euurhSSx6WmV693QUVvezHc", 2020-11-09T18:15:53.664 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.664 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.665 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.665 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.665 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.665 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945579.395388_81.1", 2020-11-09T18:15:53.666 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.666 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-8:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.8", 2020-11-09T18:15:53.666 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:59.395388143Z", 2020-11-09T18:15:53.666 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.666 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.666 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.666 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.667 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.667 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_yBzYlhlGW2rcaJy91e-7eds", 2020-11-09T18:15:53.667 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.667 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.667 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.668 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.668 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.668 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945579.400026_82.1", 2020-11-09T18:15:53.668 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.668 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-8:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.8", 2020-11-09T18:15:53.669 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:59.400026920Z", 2020-11-09T18:15:53.669 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.669 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.669 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.669 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.669 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.670 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_yBzYlhlGW2rcaJy91e-7eds", 2020-11-09T18:15:53.670 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.670 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.670 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.670 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.670 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.671 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945579.402183_83.1", 2020-11-09T18:15:53.671 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.671 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-8", 2020-11-09T18:15:53.671 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:59.402183037Z", 2020-11-09T18:15:53.671 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.672 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.672 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.672 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.672 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.672 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_OGdV9KW-t2xB2NYjc8pNB2U", 2020-11-09T18:15:53.672 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.673 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.673 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.673 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.673 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.674 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945579.406495_84.1", 2020-11-09T18:15:53.674 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.674 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-8", 2020-11-09T18:15:53.674 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:59.406495328Z", 2020-11-09T18:15:53.674 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.675 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.675 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.675 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.675 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.675 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_OGdV9KW-t2xB2NYjc8pNB2U", 2020-11-09T18:15:53.675 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.676 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.676 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.676 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.676 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.677 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945579.450093_85.1", 2020-11-09T18:15:53.677 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.677 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-9:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.9", 2020-11-09T18:15:53.677 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:59.450093555Z", 2020-11-09T18:15:53.677 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.677 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.677 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.678 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.678 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.678 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_sx-mxXXsFhATfbPuJXuh1It", 2020-11-09T18:15:53.678 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.678 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.679 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.679 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.679 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.679 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.679 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945579.455372_86.1", 2020-11-09T18:15:53.679 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.680 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-9:11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.9", 2020-11-09T18:15:53.680 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:59.455372433Z", 2020-11-09T18:15:53.680 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.680 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.680 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.680 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.680 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.681 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_sx-mxXXsFhATfbPuJXuh1It", 2020-11-09T18:15:53.681 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.681 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.681 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.681 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.681 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.682 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.682 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945579.459352_87.1", 2020-11-09T18:15:53.682 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.682 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-9", 2020-11-09T18:15:53.682 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:59.459352980Z", 2020-11-09T18:15:53.682 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.683 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.683 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.683 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.683 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.683 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_CehTzP8PVTogocbVBJZMUE-", 2020-11-09T18:15:53.683 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.684 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.684 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:53.684 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.684 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.685 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945579.467305_88.1", 2020-11-09T18:15:53.685 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.685 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-9", 2020-11-09T18:15:53.685 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:12:59.467305227Z", 2020-11-09T18:15:53.685 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.685 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.685 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:53.686 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:53.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.686 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.686 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_CehTzP8PVTogocbVBJZMUE-", 2020-11-09T18:15:53.686 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.686 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.687 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.687 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.687 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.687 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.687 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945582.240772_89.1", 2020-11-09T18:15:53.687 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.688 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-7", 2020-11-09T18:15:53.688 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.240772877Z", 2020-11-09T18:15:53.688 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.688 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.688 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_euurhSSx6WmV693QUVvezHc", 2020-11-09T18:15:53.688 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.689 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.689 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_euurhSSx6WmV693QUVvezHc", 2020-11-09T18:15:53.689 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.689 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.689 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "remove" 2020-11-09T18:15:53.689 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.690 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.690 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945582.251051_91.1", 2020-11-09T18:15:53.690 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.690 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-7", 2020-11-09T18:15:53.690 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.251051719Z", 2020-11-09T18:15:53.691 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.691 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.691 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_euurhSSx6WmV693QUVvezHc", 2020-11-09T18:15:53.691 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.691 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.691 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_euurhSSx6WmV693QUVvezHc", 2020-11-09T18:15:53.692 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.692 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.692 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.692 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.692 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.693 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945582.313479_92.1", 2020-11-09T18:15:53.693 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.693 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-8", 2020-11-09T18:15:53.693 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.313479625Z", 2020-11-09T18:15:53.693 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.693 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.694 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_OGdV9KW-t2xB2NYjc8pNB2U", 2020-11-09T18:15:53.694 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.694 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.694 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_OGdV9KW-t2xB2NYjc8pNB2U", 2020-11-09T18:15:53.694 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.695 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.695 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "remove" 2020-11-09T18:15:53.695 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.695 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.695 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.695 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945582.321374_93.1", 2020-11-09T18:15:53.696 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.696 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-8", 2020-11-09T18:15:53.696 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.321374298Z", 2020-11-09T18:15:53.696 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.696 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.696 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_OGdV9KW-t2xB2NYjc8pNB2U", 2020-11-09T18:15:53.696 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.697 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.697 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_OGdV9KW-t2xB2NYjc8pNB2U", 2020-11-09T18:15:53.697 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.697 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.697 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.698 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.698 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.698 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945582.425686_94.1", 2020-11-09T18:15:53.698 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.698 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-9", 2020-11-09T18:15:53.699 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.425686180Z", 2020-11-09T18:15:53.699 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.699 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.699 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_CehTzP8PVTogocbVBJZMUE-", 2020-11-09T18:15:53.699 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.699 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.700 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_CehTzP8PVTogocbVBJZMUE-", 2020-11-09T18:15:53.700 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.700 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.700 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "remove" 2020-11-09T18:15:53.700 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.700 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:53.701 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945582.432939_95.1", 2020-11-09T18:15:53.701 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:53.701 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-9", 2020-11-09T18:15:53.701 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939430Z", 2020-11-09T18:15:53.701 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:53.702 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:53.702 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_CehTzP8PVTogocbVBJZMUE-", 2020-11-09T18:15:53.702 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:53.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.702 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:53.702 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_CehTzP8PVTogocbVBJZMUE-", 2020-11-09T18:15:53.702 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 2 2020-11-09T18:15:53.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:53.703 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:53.703 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:53.703 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.703 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.703 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:53.704 INFO:teuthology.orchestra.run.smithi078.stdout:] 2020-11-09T18:15:53.705 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'bbf9f070-21b9-4940-95e9-d031ba1c647f', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:53.705 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'bbf9f070-21b9-4940-95e9-d031ba1c647f', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:53.706 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period bbf9f070-21b9-4940-95e9-d031ba1c647f --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:15:53.748 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:15:53.749 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:15:53.990 INFO:teuthology.orchestra.run.smithi051.stdout:[ 2020-11-09T18:15:53.990 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:53.990 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945680.562349_152.1", 2020-11-09T18:15:53.990 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.991 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-42:4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc.4649.1", 2020-11-09T18:15:53.991 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:14:40.562349167Z", 2020-11-09T18:15:53.991 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:53.991 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:53.991 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:53.991 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:53.992 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:53.992 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:53.992 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_GEhFoWhTZz2s9yC9WN91fjc", 2020-11-09T18:15:53.992 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:53.992 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:53.992 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:53.993 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:53.993 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:53.993 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:53.993 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:53.993 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:53.993 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945680.567866_153.1", 2020-11-09T18:15:53.994 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:53.995 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-42:4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc.4649.1", 2020-11-09T18:15:53.995 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:14:40.567866437Z", 2020-11-09T18:15:53.995 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:53.995 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:53.996 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:53.996 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:53.996 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:53.996 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:53.997 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_GEhFoWhTZz2s9yC9WN91fjc", 2020-11-09T18:15:53.997 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:53.997 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:53.997 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:53.998 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:53.998 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:53.998 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:53.999 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:53.999 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:53.999 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945680.570048_154.1", 2020-11-09T18:15:53.999 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:54.000 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-42", 2020-11-09T18:15:54.000 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:14:40.570048528Z", 2020-11-09T18:15:54.000 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:54.000 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:54.001 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:54.001 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:54.001 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:54.001 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:54.002 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_hFkMyDWeU8TWQK9zlQK3WTR", 2020-11-09T18:15:54.002 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:54.002 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:54.002 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:54.003 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:54.003 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:54.003 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:54.003 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:54.004 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:54.004 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945680.575646_155.1", 2020-11-09T18:15:54.004 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:54.004 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-42", 2020-11-09T18:15:54.005 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:14:40.575646871Z", 2020-11-09T18:15:54.005 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:54.005 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:54.005 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:54.006 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:54.006 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:54.006 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:54.007 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_hFkMyDWeU8TWQK9zlQK3WTR", 2020-11-09T18:15:54.007 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:54.007 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:54.008 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:54.008 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:54.008 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:54.008 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:54.009 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:54.009 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:54.009 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945680.627136_156.1", 2020-11-09T18:15:54.010 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:54.010 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-43:4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc.4649.2", 2020-11-09T18:15:54.010 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:14:40.627136387Z", 2020-11-09T18:15:54.010 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:54.010 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:54.011 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:54.011 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:54.011 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:54.011 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:54.011 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_-hy1rG5z1TdHJQfEPyJr45Z", 2020-11-09T18:15:54.011 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:54.012 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:54.012 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:54.012 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:54.012 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:54.012 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:54.012 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:54.012 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:54.013 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945680.631397_157.1", 2020-11-09T18:15:54.013 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:54.013 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-43:4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc.4649.2", 2020-11-09T18:15:54.013 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:14:40.631397430Z", 2020-11-09T18:15:54.013 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:54.013 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:54.013 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:54.013 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:54.014 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:54.014 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:54.014 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_-hy1rG5z1TdHJQfEPyJr45Z", 2020-11-09T18:15:54.014 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:54.014 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:54.014 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:54.014 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:54.015 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:54.015 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:54.015 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:54.015 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:54.015 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945680.633297_158.1", 2020-11-09T18:15:54.015 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:54.015 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-43", 2020-11-09T18:15:54.016 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:14:40.633297989Z", 2020-11-09T18:15:54.016 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:54.016 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:54.016 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:54.016 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:54.016 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:54.017 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:54.017 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_bWt1ThWeCRRKM2smwWkqAPS", 2020-11-09T18:15:54.017 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:54.017 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:54.017 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:54.018 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:54.018 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:54.018 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:54.018 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:54.018 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:54.018 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945680.637050_159.1", 2020-11-09T18:15:54.019 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:54.019 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-43", 2020-11-09T18:15:54.019 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:14:40.637050782Z", 2020-11-09T18:15:54.019 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:54.019 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:54.019 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:54.019 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:54.020 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:54.020 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:54.020 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_bWt1ThWeCRRKM2smwWkqAPS", 2020-11-09T18:15:54.020 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:54.020 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:54.020 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:54.023 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:54.023 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:54.023 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:54.023 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:54.023 INFO:teuthology.orchestra.run.smithi051.stdout:] 2020-11-09T18:15:54.025 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'bbf9f070-21b9-4940-95e9-d031ba1c647f', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:54.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'bbf9f070-21b9-4940-95e9-d031ba1c647f', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:54.027 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period bbf9f070-21b9-4940-95e9-d031ba1c647f --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:15:54.067 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:15:54.067 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:15:54.273 INFO:teuthology.orchestra.run.smithi078.stdout:[ 2020-11-09T18:15:54.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:54.273 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945680.562349_152.1", 2020-11-09T18:15:54.273 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:54.274 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-42:4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc.4649.1", 2020-11-09T18:15:54.274 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:14:40.562349167Z", 2020-11-09T18:15:54.274 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:54.274 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:54.274 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:54.274 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:54.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:54.275 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:54.275 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_GEhFoWhTZz2s9yC9WN91fjc", 2020-11-09T18:15:54.275 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:54.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:54.275 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:54.275 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:54.275 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:54.276 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:54.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:54.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:54.276 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945680.567866_153.1", 2020-11-09T18:15:54.276 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:54.276 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-42:4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc.4649.1", 2020-11-09T18:15:54.276 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:14:40.567866437Z", 2020-11-09T18:15:54.277 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:54.277 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:54.277 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:54.277 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:54.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:54.277 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:54.277 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_GEhFoWhTZz2s9yC9WN91fjc", 2020-11-09T18:15:54.278 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:54.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:54.278 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:54.278 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:54.278 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:54.278 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:54.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:54.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:54.279 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945680.570048_154.1", 2020-11-09T18:15:54.279 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:54.279 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-42", 2020-11-09T18:15:54.279 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:14:40.570048528Z", 2020-11-09T18:15:54.279 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:54.279 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:54.279 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:54.280 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:54.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:54.280 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:54.280 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_hFkMyDWeU8TWQK9zlQK3WTR", 2020-11-09T18:15:54.280 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:54.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:54.280 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:54.281 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:54.281 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:54.281 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:54.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:54.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:54.281 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945680.575646_155.1", 2020-11-09T18:15:54.281 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:54.281 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-42", 2020-11-09T18:15:54.282 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:14:40.575646871Z", 2020-11-09T18:15:54.282 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:54.282 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:54.282 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:54.282 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:54.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:54.282 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:54.283 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_hFkMyDWeU8TWQK9zlQK3WTR", 2020-11-09T18:15:54.283 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:54.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:54.283 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:54.283 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:54.283 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:54.283 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:54.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:54.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:54.284 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945680.627136_156.1", 2020-11-09T18:15:54.284 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:54.284 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-43:4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc.4649.2", 2020-11-09T18:15:54.284 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:14:40.627136387Z", 2020-11-09T18:15:54.284 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:54.284 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:54.285 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:54.285 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:54.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:54.285 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:54.285 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_-hy1rG5z1TdHJQfEPyJr45Z", 2020-11-09T18:15:54.285 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:54.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:54.286 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:54.286 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:54.286 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:54.286 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:54.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:54.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:54.286 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945680.631397_157.1", 2020-11-09T18:15:54.287 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:54.287 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-43:4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc.4649.2", 2020-11-09T18:15:54.287 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:14:40.631397430Z", 2020-11-09T18:15:54.287 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:54.287 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:54.287 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:54.287 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:54.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:54.288 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:54.288 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_-hy1rG5z1TdHJQfEPyJr45Z", 2020-11-09T18:15:54.288 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:54.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:54.288 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:54.288 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:54.288 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:54.289 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:54.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:54.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:54.289 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945680.633297_158.1", 2020-11-09T18:15:54.289 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:54.289 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-43", 2020-11-09T18:15:54.289 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:14:40.633297989Z", 2020-11-09T18:15:54.290 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:54.290 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:54.290 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:54.290 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:54.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:54.290 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:54.290 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_bWt1ThWeCRRKM2smwWkqAPS", 2020-11-09T18:15:54.290 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:54.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:54.291 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:54.291 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:54.291 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:54.291 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:54.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:54.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:54.292 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945680.637050_159.1", 2020-11-09T18:15:54.292 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:54.292 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-43", 2020-11-09T18:15:54.292 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:14:40.637050782Z", 2020-11-09T18:15:54.292 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:54.292 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:54.292 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:54.293 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:54.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:54.293 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:54.293 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_bWt1ThWeCRRKM2smwWkqAPS", 2020-11-09T18:15:54.293 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:54.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:54.293 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:54.307 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:54.307 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:54.308 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:54.308 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:54.308 INFO:teuthology.orchestra.run.smithi078.stdout:] 2020-11-09T18:15:54.309 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'bbf9f070-21b9-4940-95e9-d031ba1c647f', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:54.310 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'bbf9f070-21b9-4940-95e9-d031ba1c647f', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:54.310 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period bbf9f070-21b9-4940-95e9-d031ba1c647f --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:15:54.348 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:15:54.348 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:15:54.587 INFO:teuthology.orchestra.run.smithi051.stdout:[ 2020-11-09T18:15:54.587 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:54.587 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945680.562349_152.1", 2020-11-09T18:15:54.587 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:54.588 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-42:4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc.4649.1", 2020-11-09T18:15:54.588 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:14:40.562349167Z", 2020-11-09T18:15:54.588 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:54.588 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:54.588 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:54.588 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:54.588 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:54.589 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:54.589 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_GEhFoWhTZz2s9yC9WN91fjc", 2020-11-09T18:15:54.589 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:54.589 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:54.589 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:54.589 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:54.589 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:54.589 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:54.590 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:54.590 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:54.590 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945680.567866_153.1", 2020-11-09T18:15:54.590 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:54.590 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-42:4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc.4649.1", 2020-11-09T18:15:54.590 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:14:40.567866437Z", 2020-11-09T18:15:54.590 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:54.591 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:54.591 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:54.591 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:54.591 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:54.591 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:54.591 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_GEhFoWhTZz2s9yC9WN91fjc", 2020-11-09T18:15:54.591 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:54.592 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:54.592 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:54.592 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:54.592 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:54.592 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:54.592 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:54.592 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:54.593 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945680.570048_154.1", 2020-11-09T18:15:54.593 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:54.593 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-42", 2020-11-09T18:15:54.593 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:14:40.570048528Z", 2020-11-09T18:15:54.593 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:54.594 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:54.594 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:54.594 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:54.594 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:54.594 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:54.594 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_hFkMyDWeU8TWQK9zlQK3WTR", 2020-11-09T18:15:54.594 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:54.595 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:54.595 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:54.595 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:54.595 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:54.595 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:54.595 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:54.595 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:54.595 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945680.575646_155.1", 2020-11-09T18:15:54.596 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:54.596 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-42", 2020-11-09T18:15:54.596 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:14:40.575646871Z", 2020-11-09T18:15:54.596 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:54.596 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:54.596 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:54.596 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:54.597 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:54.597 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:54.597 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_hFkMyDWeU8TWQK9zlQK3WTR", 2020-11-09T18:15:54.597 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:54.597 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:54.597 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:54.597 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:54.597 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:54.598 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:54.598 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:54.598 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:54.598 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945680.627136_156.1", 2020-11-09T18:15:54.598 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:54.598 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-43:4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc.4649.2", 2020-11-09T18:15:54.598 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:14:40.627136387Z", 2020-11-09T18:15:54.599 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:54.599 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:54.599 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:54.599 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:54.599 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:54.599 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:54.599 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_-hy1rG5z1TdHJQfEPyJr45Z", 2020-11-09T18:15:54.600 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:54.600 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:54.600 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:54.600 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:54.600 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:54.600 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:54.600 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:54.600 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:54.601 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945680.631397_157.1", 2020-11-09T18:15:54.601 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:54.601 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-43:4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc.4649.2", 2020-11-09T18:15:54.601 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:14:40.631397430Z", 2020-11-09T18:15:54.601 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:54.601 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:54.601 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:54.601 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:54.602 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:54.602 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:54.602 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_-hy1rG5z1TdHJQfEPyJr45Z", 2020-11-09T18:15:54.602 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:54.602 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:54.602 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:54.602 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:54.603 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:54.603 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:54.603 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:54.603 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:54.603 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945680.633297_158.1", 2020-11-09T18:15:54.603 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:54.603 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-43", 2020-11-09T18:15:54.604 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:14:40.633297989Z", 2020-11-09T18:15:54.604 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:54.604 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:54.604 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:54.604 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:54.604 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:54.604 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:54.604 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_bWt1ThWeCRRKM2smwWkqAPS", 2020-11-09T18:15:54.605 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:54.605 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:54.605 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:54.605 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:54.605 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:54.605 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:54.605 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:54.606 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:54.606 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945680.637050_159.1", 2020-11-09T18:15:54.606 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:54.606 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-43", 2020-11-09T18:15:54.606 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:14:40.637050782Z", 2020-11-09T18:15:54.606 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:54.606 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:54.606 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:54.607 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:54.607 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:54.607 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:54.607 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_bWt1ThWeCRRKM2smwWkqAPS", 2020-11-09T18:15:54.607 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:54.607 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:54.607 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:54.620 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:54.621 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:54.621 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:54.621 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:54.621 INFO:teuthology.orchestra.run.smithi051.stdout:] 2020-11-09T18:15:54.623 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'bbf9f070-21b9-4940-95e9-d031ba1c647f', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:54.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'bbf9f070-21b9-4940-95e9-d031ba1c647f', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:54.624 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period bbf9f070-21b9-4940-95e9-d031ba1c647f --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:15:54.663 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:15:54.663 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:15:54.879 INFO:teuthology.orchestra.run.smithi078.stdout:[ 2020-11-09T18:15:54.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:54.879 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945680.562349_152.1", 2020-11-09T18:15:54.880 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:54.880 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-42:4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc.4649.1", 2020-11-09T18:15:54.880 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:14:40.562349167Z", 2020-11-09T18:15:54.880 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:54.880 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:54.880 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:54.880 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:54.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:54.881 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:54.881 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_GEhFoWhTZz2s9yC9WN91fjc", 2020-11-09T18:15:54.881 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:54.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:54.881 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:54.881 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:54.882 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:54.882 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:54.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:54.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:54.882 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945680.567866_153.1", 2020-11-09T18:15:54.882 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:54.882 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-42:4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc.4649.1", 2020-11-09T18:15:54.883 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:14:40.567866437Z", 2020-11-09T18:15:54.883 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:54.883 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:54.883 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:54.883 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:54.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:54.883 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:54.884 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_GEhFoWhTZz2s9yC9WN91fjc", 2020-11-09T18:15:54.884 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:54.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:54.884 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:54.884 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:54.884 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:54.884 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:54.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:54.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:54.885 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945680.570048_154.1", 2020-11-09T18:15:54.885 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:54.885 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-42", 2020-11-09T18:15:54.885 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:14:40.570048528Z", 2020-11-09T18:15:54.885 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:54.885 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:54.886 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:54.886 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:54.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:54.886 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:54.886 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_hFkMyDWeU8TWQK9zlQK3WTR", 2020-11-09T18:15:54.886 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:54.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:54.887 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:54.887 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:54.887 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:54.887 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:54.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:54.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:54.887 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945680.575646_155.1", 2020-11-09T18:15:54.888 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:54.888 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-42", 2020-11-09T18:15:54.888 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:14:40.575646871Z", 2020-11-09T18:15:54.888 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:54.888 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:54.888 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:54.888 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:54.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:54.889 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:54.889 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_hFkMyDWeU8TWQK9zlQK3WTR", 2020-11-09T18:15:54.889 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:54.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:54.889 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:54.889 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:54.889 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:54.890 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:54.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:54.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:54.890 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945680.627136_156.1", 2020-11-09T18:15:54.890 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:54.890 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-43:4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc.4649.2", 2020-11-09T18:15:54.890 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:14:40.627136387Z", 2020-11-09T18:15:54.890 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:54.891 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:54.891 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:54.891 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:54.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:54.891 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:54.891 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_-hy1rG5z1TdHJQfEPyJr45Z", 2020-11-09T18:15:54.891 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:54.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:54.892 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:54.892 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:54.892 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:54.892 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:54.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:54.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:54.893 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945680.631397_157.1", 2020-11-09T18:15:54.893 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:54.893 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-43:4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc.4649.2", 2020-11-09T18:15:54.893 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:14:40.631397430Z", 2020-11-09T18:15:54.893 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:54.893 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:54.893 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:54.893 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:54.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:54.894 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:54.894 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_-hy1rG5z1TdHJQfEPyJr45Z", 2020-11-09T18:15:54.894 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:54.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:54.894 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:54.894 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:54.895 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:54.895 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:54.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:54.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:54.895 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945680.633297_158.1", 2020-11-09T18:15:54.895 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:54.895 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-43", 2020-11-09T18:15:54.896 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:14:40.633297989Z", 2020-11-09T18:15:54.896 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:54.896 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:54.896 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:54.896 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:54.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:54.896 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:54.896 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_bWt1ThWeCRRKM2smwWkqAPS", 2020-11-09T18:15:54.897 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:54.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:54.897 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:54.897 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:54.897 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:54.897 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:54.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:54.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:54.898 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945680.637050_159.1", 2020-11-09T18:15:54.898 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:54.898 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-43", 2020-11-09T18:15:54.898 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:14:40.637050782Z", 2020-11-09T18:15:54.898 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:54.898 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:54.898 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:54.899 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:54.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:54.899 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:54.899 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_bWt1ThWeCRRKM2smwWkqAPS", 2020-11-09T18:15:54.899 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:54.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:54.899 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:54.909 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:54.909 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:54.910 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:54.910 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:54.910 INFO:teuthology.orchestra.run.smithi078.stdout:] 2020-11-09T18:15:54.911 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'f6258aba-300a-4d5d-ad93-a7c198a0e0a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:54.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'f6258aba-300a-4d5d-ad93-a7c198a0e0a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:54.912 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period f6258aba-300a-4d5d-ad93-a7c198a0e0a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:15:54.951 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:15:54.951 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:15:55.199 INFO:teuthology.orchestra.run.smithi051.stdout:[ 2020-11-09T18:15:55.199 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:55.199 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945741.824477_167.1", 2020-11-09T18:15:55.200 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:55.200 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-44:11f516a7-3288-49a6-9a89-b45ac1dae91a.4545.1", 2020-11-09T18:15:55.200 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:15:41.824477020Z", 2020-11-09T18:15:55.200 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:55.201 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:55.201 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:55.201 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:55.202 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:55.202 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:55.202 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_HWkiUHLzheT-uW_BN9huNr3", 2020-11-09T18:15:55.202 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:55.203 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:55.203 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:55.203 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:55.204 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:55.204 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:55.204 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:55.204 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:55.205 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945741.829560_168.1", 2020-11-09T18:15:55.205 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:55.205 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-44:11f516a7-3288-49a6-9a89-b45ac1dae91a.4545.1", 2020-11-09T18:15:55.205 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:15:41.829560884Z", 2020-11-09T18:15:55.206 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:55.206 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:55.206 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:55.207 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:55.207 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:55.207 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:55.207 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_HWkiUHLzheT-uW_BN9huNr3", 2020-11-09T18:15:55.208 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:55.208 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:55.208 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:55.208 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:55.209 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:55.209 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:55.209 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:55.210 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:55.210 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945741.832197_169.1", 2020-11-09T18:15:55.210 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:55.210 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-44", 2020-11-09T18:15:55.211 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:15:41.832197849Z", 2020-11-09T18:15:55.211 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:55.211 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:55.211 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:55.212 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:55.212 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:55.212 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:55.213 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_05qjX7fK0wyD_LgFnXyvwBE", 2020-11-09T18:15:55.213 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:55.213 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:55.213 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:55.214 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:55.214 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:55.214 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:55.215 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:55.215 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:55.215 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945741.837831_170.1", 2020-11-09T18:15:55.215 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:55.216 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-44", 2020-11-09T18:15:55.216 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:15:41.837831337Z", 2020-11-09T18:15:55.216 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:55.217 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:55.217 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:55.217 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:55.217 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:55.217 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:55.218 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_05qjX7fK0wyD_LgFnXyvwBE", 2020-11-09T18:15:55.218 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:55.218 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:55.218 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:55.219 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:55.219 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:55.219 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:55.219 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:55.219 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:55.219 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945741.886929_171.1", 2020-11-09T18:15:55.220 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:55.220 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-45:11f516a7-3288-49a6-9a89-b45ac1dae91a.4545.2", 2020-11-09T18:15:55.220 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:15:41.886929099Z", 2020-11-09T18:15:55.220 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:55.220 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:55.221 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:55.221 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:55.221 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:55.221 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:55.221 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_AX1zZnHeh4-IDrumhjksoqE", 2020-11-09T18:15:55.221 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:55.222 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:55.222 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:55.222 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:55.222 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:55.222 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:55.223 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:55.223 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:55.223 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945741.892121_172.1", 2020-11-09T18:15:55.223 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:55.223 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-45:11f516a7-3288-49a6-9a89-b45ac1dae91a.4545.2", 2020-11-09T18:15:55.224 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:15:41.892121499Z", 2020-11-09T18:15:55.224 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:55.224 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:55.224 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:55.224 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:55.224 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:55.224 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:55.225 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_AX1zZnHeh4-IDrumhjksoqE", 2020-11-09T18:15:55.225 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:55.225 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:55.225 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:55.225 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:55.225 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:55.225 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:55.226 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:55.226 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:55.226 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945741.894552_173.1", 2020-11-09T18:15:55.226 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:55.226 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-45", 2020-11-09T18:15:55.226 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:15:41.894552930Z", 2020-11-09T18:15:55.226 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:55.227 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:55.227 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:55.227 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:55.227 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:55.227 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:55.227 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_6XlCxsMbRQlyzSvVHKszd2P", 2020-11-09T18:15:55.227 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:55.227 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:55.228 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:55.228 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:55.228 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:55.228 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:55.228 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:55.228 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:55.228 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945741.900663_174.1", 2020-11-09T18:15:55.229 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:55.229 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-45", 2020-11-09T18:15:55.229 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:15:41.900663488Z", 2020-11-09T18:15:55.229 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:55.229 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:55.229 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:55.229 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:55.229 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:55.230 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:55.230 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_6XlCxsMbRQlyzSvVHKszd2P", 2020-11-09T18:15:55.230 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:55.230 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:55.230 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:55.233 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:55.233 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:55.233 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:55.233 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:55.234 INFO:teuthology.orchestra.run.smithi051.stdout:] 2020-11-09T18:15:55.236 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'f6258aba-300a-4d5d-ad93-a7c198a0e0a1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:55.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'f6258aba-300a-4d5d-ad93-a7c198a0e0a1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:55.236 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period f6258aba-300a-4d5d-ad93-a7c198a0e0a1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:15:55.276 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:15:55.276 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:15:55.512 INFO:teuthology.orchestra.run.smithi078.stdout:[ 2020-11-09T18:15:55.513 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:55.513 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945741.824477_167.1", 2020-11-09T18:15:55.513 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:55.513 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-44:11f516a7-3288-49a6-9a89-b45ac1dae91a.4545.1", 2020-11-09T18:15:55.514 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:15:41.824477020Z", 2020-11-09T18:15:55.514 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:55.514 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:55.514 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:55.514 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:55.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:55.514 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:55.515 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_HWkiUHLzheT-uW_BN9huNr3", 2020-11-09T18:15:55.515 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:55.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:55.515 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:55.515 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:55.515 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:55.515 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:55.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:55.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:55.516 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945741.829560_168.1", 2020-11-09T18:15:55.516 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:55.516 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-44:11f516a7-3288-49a6-9a89-b45ac1dae91a.4545.1", 2020-11-09T18:15:55.516 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:15:41.829560884Z", 2020-11-09T18:15:55.516 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:55.517 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:55.517 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:55.517 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:55.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:55.517 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:55.517 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_HWkiUHLzheT-uW_BN9huNr3", 2020-11-09T18:15:55.517 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:55.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:55.518 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:55.518 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:55.518 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:55.518 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:55.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:55.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:55.518 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945741.832197_169.1", 2020-11-09T18:15:55.519 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:55.519 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-44", 2020-11-09T18:15:55.519 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:15:41.832197849Z", 2020-11-09T18:15:55.519 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:55.519 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:55.519 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:55.519 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:55.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:55.520 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:55.520 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_05qjX7fK0wyD_LgFnXyvwBE", 2020-11-09T18:15:55.520 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:55.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:55.520 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:55.520 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:55.520 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:55.521 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:55.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:55.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:55.521 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945741.837831_170.1", 2020-11-09T18:15:55.521 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:55.521 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-44", 2020-11-09T18:15:55.521 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:15:41.837831337Z", 2020-11-09T18:15:55.522 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:55.522 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:55.522 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:55.522 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:55.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:55.522 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:55.522 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_05qjX7fK0wyD_LgFnXyvwBE", 2020-11-09T18:15:55.522 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:55.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:55.523 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:55.523 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:55.523 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:55.523 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:55.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:55.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:55.524 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945741.886929_171.1", 2020-11-09T18:15:55.524 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:55.524 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-45:11f516a7-3288-49a6-9a89-b45ac1dae91a.4545.2", 2020-11-09T18:15:55.524 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:15:41.886929099Z", 2020-11-09T18:15:55.524 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:55.524 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:55.524 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:55.524 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:55.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:55.525 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:55.525 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_AX1zZnHeh4-IDrumhjksoqE", 2020-11-09T18:15:55.525 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:55.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:55.525 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:55.525 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:55.526 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:55.526 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:55.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:55.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:55.526 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945741.892121_172.1", 2020-11-09T18:15:55.526 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:55.527 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-45:11f516a7-3288-49a6-9a89-b45ac1dae91a.4545.2", 2020-11-09T18:15:55.527 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:15:41.892121499Z", 2020-11-09T18:15:55.527 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:55.527 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:55.527 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:55.527 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:55.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:55.527 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:55.528 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_AX1zZnHeh4-IDrumhjksoqE", 2020-11-09T18:15:55.528 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:55.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:55.528 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:55.528 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:55.528 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:55.528 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:55.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:55.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:55.529 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945741.894552_173.1", 2020-11-09T18:15:55.529 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:55.529 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-45", 2020-11-09T18:15:55.529 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:15:41.894552930Z", 2020-11-09T18:15:55.529 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:55.530 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:55.530 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:55.530 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:55.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:55.530 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:55.530 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_6XlCxsMbRQlyzSvVHKszd2P", 2020-11-09T18:15:55.530 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:55.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:55.531 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:55.531 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:55.531 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:55.531 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:55.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:55.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:55.531 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945741.900663_174.1", 2020-11-09T18:15:55.532 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:55.532 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-45", 2020-11-09T18:15:55.532 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:15:41.900663488Z", 2020-11-09T18:15:55.532 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:55.532 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:55.532 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:55.532 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:55.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:55.533 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:55.533 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_6XlCxsMbRQlyzSvVHKszd2P", 2020-11-09T18:15:55.533 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:55.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:55.533 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:55.545 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:55.545 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:55.545 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:55.546 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:55.546 INFO:teuthology.orchestra.run.smithi078.stdout:] 2020-11-09T18:15:55.548 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'f6258aba-300a-4d5d-ad93-a7c198a0e0a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:55.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'f6258aba-300a-4d5d-ad93-a7c198a0e0a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:55.549 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period f6258aba-300a-4d5d-ad93-a7c198a0e0a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:15:55.587 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:15:55.588 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:15:55.832 INFO:teuthology.orchestra.run.smithi051.stdout:[ 2020-11-09T18:15:55.833 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:55.833 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945741.824477_167.1", 2020-11-09T18:15:55.833 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:55.833 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-44:11f516a7-3288-49a6-9a89-b45ac1dae91a.4545.1", 2020-11-09T18:15:55.834 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:15:41.824477020Z", 2020-11-09T18:15:55.834 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:55.834 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:55.835 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:55.835 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:55.835 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:55.835 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:55.835 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_HWkiUHLzheT-uW_BN9huNr3", 2020-11-09T18:15:55.836 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:55.836 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:55.836 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:55.836 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:55.837 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:55.837 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:55.837 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:55.837 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:55.838 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945741.829560_168.1", 2020-11-09T18:15:55.838 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:55.838 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-44:11f516a7-3288-49a6-9a89-b45ac1dae91a.4545.1", 2020-11-09T18:15:55.838 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:15:41.829560884Z", 2020-11-09T18:15:55.839 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:55.839 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:55.839 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:55.839 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:55.839 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:55.840 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:55.840 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_HWkiUHLzheT-uW_BN9huNr3", 2020-11-09T18:15:55.840 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:55.840 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:55.840 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:55.840 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:55.840 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:55.841 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:55.841 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:55.841 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:55.841 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945741.832197_169.1", 2020-11-09T18:15:55.841 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:55.841 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-44", 2020-11-09T18:15:55.841 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:15:41.832197849Z", 2020-11-09T18:15:55.842 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:55.842 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:55.842 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:55.842 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:55.842 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:55.843 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:55.843 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_05qjX7fK0wyD_LgFnXyvwBE", 2020-11-09T18:15:55.843 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:55.843 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:55.843 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:55.844 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:55.844 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:55.844 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:55.844 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:55.844 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:55.845 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945741.837831_170.1", 2020-11-09T18:15:55.845 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:55.845 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-44", 2020-11-09T18:15:55.845 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:15:41.837831337Z", 2020-11-09T18:15:55.846 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:55.846 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:55.846 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:55.846 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:55.847 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:55.847 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:55.847 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_05qjX7fK0wyD_LgFnXyvwBE", 2020-11-09T18:15:55.848 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:55.848 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:55.848 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:55.848 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:55.849 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:55.849 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:55.849 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:55.849 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:55.850 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945741.886929_171.1", 2020-11-09T18:15:55.850 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:55.850 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-45:11f516a7-3288-49a6-9a89-b45ac1dae91a.4545.2", 2020-11-09T18:15:55.850 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:15:41.886929099Z", 2020-11-09T18:15:55.851 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:55.851 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:55.851 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:55.851 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:55.851 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:55.852 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:55.852 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_AX1zZnHeh4-IDrumhjksoqE", 2020-11-09T18:15:55.852 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:55.853 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:55.853 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:55.853 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:55.853 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:55.853 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:55.854 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:55.854 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:55.854 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945741.892121_172.1", 2020-11-09T18:15:55.854 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket.instance", 2020-11-09T18:15:55.854 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-45:11f516a7-3288-49a6-9a89-b45ac1dae91a.4545.2", 2020-11-09T18:15:55.854 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:15:41.892121499Z", 2020-11-09T18:15:55.855 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:55.855 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:55.855 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:55.855 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:55.855 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:55.855 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:55.855 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_AX1zZnHeh4-IDrumhjksoqE", 2020-11-09T18:15:55.856 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:55.856 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:55.856 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:55.856 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:55.856 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:55.856 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:55.856 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:55.857 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:55.857 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945741.894552_173.1", 2020-11-09T18:15:55.857 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:55.857 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-45", 2020-11-09T18:15:55.857 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:15:41.894552930Z", 2020-11-09T18:15:55.857 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:55.857 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:55.857 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:55.858 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:55.858 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:55.858 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:55.858 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_6XlCxsMbRQlyzSvVHKszd2P", 2020-11-09T18:15:55.858 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:55.858 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:55.858 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:55.859 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "write" 2020-11-09T18:15:55.859 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:55.859 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:55.859 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:55.859 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:15:55.859 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "1_1604945741.900663_174.1", 2020-11-09T18:15:55.859 INFO:teuthology.orchestra.run.smithi051.stdout: "section": "bucket", 2020-11-09T18:15:55.860 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "ymzeui-45", 2020-11-09T18:15:55.860 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:15:41.900663488Z", 2020-11-09T18:15:55.860 INFO:teuthology.orchestra.run.smithi051.stdout: "data": { 2020-11-09T18:15:55.860 INFO:teuthology.orchestra.run.smithi051.stdout: "read_version": { 2020-11-09T18:15:55.860 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "", 2020-11-09T18:15:55.860 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 0 2020-11-09T18:15:55.860 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:55.861 INFO:teuthology.orchestra.run.smithi051.stdout: "write_version": { 2020-11-09T18:15:55.861 INFO:teuthology.orchestra.run.smithi051.stdout: "tag": "_6XlCxsMbRQlyzSvVHKszd2P", 2020-11-09T18:15:55.861 INFO:teuthology.orchestra.run.smithi051.stdout: "ver": 1 2020-11-09T18:15:55.861 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:15:55.862 INFO:teuthology.orchestra.run.smithi051.stdout: "status": { 2020-11-09T18:15:55.866 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "complete" 2020-11-09T18:15:55.866 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:55.866 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:55.866 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:15:55.867 INFO:teuthology.orchestra.run.smithi051.stdout:] 2020-11-09T18:15:55.867 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'f6258aba-300a-4d5d-ad93-a7c198a0e0a1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:55.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'f6258aba-300a-4d5d-ad93-a7c198a0e0a1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:55.868 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period f6258aba-300a-4d5d-ad93-a7c198a0e0a1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:15:55.908 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:15:55.908 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:15:56.123 INFO:teuthology.orchestra.run.smithi078.stdout:[ 2020-11-09T18:15:56.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:56.124 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945741.824477_167.1", 2020-11-09T18:15:56.124 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:56.124 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-44:11f516a7-3288-49a6-9a89-b45ac1dae91a.4545.1", 2020-11-09T18:15:56.124 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:15:41.824477020Z", 2020-11-09T18:15:56.124 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:56.125 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:56.125 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:56.125 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:56.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:56.125 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:56.125 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_HWkiUHLzheT-uW_BN9huNr3", 2020-11-09T18:15:56.126 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:56.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:56.126 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:56.126 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:56.126 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:56.127 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:56.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:56.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:56.127 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945741.829560_168.1", 2020-11-09T18:15:56.127 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:56.128 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-44:11f516a7-3288-49a6-9a89-b45ac1dae91a.4545.1", 2020-11-09T18:15:56.128 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:15:41.829560884Z", 2020-11-09T18:15:56.128 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:56.128 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:56.128 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:56.129 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:56.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:56.129 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:56.129 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_HWkiUHLzheT-uW_BN9huNr3", 2020-11-09T18:15:56.129 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:56.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:56.129 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:56.130 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:56.130 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:56.130 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:56.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:56.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:56.130 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945741.832197_169.1", 2020-11-09T18:15:56.131 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:56.131 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-44", 2020-11-09T18:15:56.131 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:15:41.832197849Z", 2020-11-09T18:15:56.131 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:56.131 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:56.131 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:56.131 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:56.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:56.132 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:56.132 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_05qjX7fK0wyD_LgFnXyvwBE", 2020-11-09T18:15:56.132 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:56.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:56.132 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:56.132 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:56.133 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:56.133 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:56.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:56.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:56.133 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945741.837831_170.1", 2020-11-09T18:15:56.133 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:56.133 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-44", 2020-11-09T18:15:56.133 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:15:41.837831337Z", 2020-11-09T18:15:56.134 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:56.134 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:56.134 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:56.134 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:56.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:56.134 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:56.135 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_05qjX7fK0wyD_LgFnXyvwBE", 2020-11-09T18:15:56.135 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:56.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:56.135 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:56.135 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:56.135 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:56.136 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:56.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:56.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:56.136 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945741.886929_171.1", 2020-11-09T18:15:56.136 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:56.136 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-45:11f516a7-3288-49a6-9a89-b45ac1dae91a.4545.2", 2020-11-09T18:15:56.136 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:15:41.886929099Z", 2020-11-09T18:15:56.137 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:56.137 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:56.137 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:56.137 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:56.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:56.137 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:56.138 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_AX1zZnHeh4-IDrumhjksoqE", 2020-11-09T18:15:56.138 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:56.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:56.138 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:56.138 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:56.138 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:56.138 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:56.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:56.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:56.139 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945741.892121_172.1", 2020-11-09T18:15:56.139 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket.instance", 2020-11-09T18:15:56.139 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-45:11f516a7-3288-49a6-9a89-b45ac1dae91a.4545.2", 2020-11-09T18:15:56.139 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:15:41.892121499Z", 2020-11-09T18:15:56.139 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:56.140 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:56.140 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:56.140 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:56.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:56.140 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:56.140 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_AX1zZnHeh4-IDrumhjksoqE", 2020-11-09T18:15:56.141 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:56.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:56.141 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:56.141 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:56.141 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:56.141 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:56.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:56.142 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:56.142 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945741.894552_173.1", 2020-11-09T18:15:56.142 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:56.142 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-45", 2020-11-09T18:15:56.142 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:15:41.894552930Z", 2020-11-09T18:15:56.142 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:56.142 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:56.143 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:56.143 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:56.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:56.143 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:56.143 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_6XlCxsMbRQlyzSvVHKszd2P", 2020-11-09T18:15:56.143 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:56.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:56.144 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:56.144 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "write" 2020-11-09T18:15:56.144 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:56.144 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:56.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:56.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:15:56.144 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1_1604945741.900663_174.1", 2020-11-09T18:15:56.144 INFO:teuthology.orchestra.run.smithi078.stdout: "section": "bucket", 2020-11-09T18:15:56.145 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ymzeui-45", 2020-11-09T18:15:56.145 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:15:41.900663488Z", 2020-11-09T18:15:56.145 INFO:teuthology.orchestra.run.smithi078.stdout: "data": { 2020-11-09T18:15:56.145 INFO:teuthology.orchestra.run.smithi078.stdout: "read_version": { 2020-11-09T18:15:56.145 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "", 2020-11-09T18:15:56.145 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 0 2020-11-09T18:15:56.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:56.146 INFO:teuthology.orchestra.run.smithi078.stdout: "write_version": { 2020-11-09T18:15:56.146 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "_6XlCxsMbRQlyzSvVHKszd2P", 2020-11-09T18:15:56.146 INFO:teuthology.orchestra.run.smithi078.stdout: "ver": 1 2020-11-09T18:15:56.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:15:56.146 INFO:teuthology.orchestra.run.smithi078.stdout: "status": { 2020-11-09T18:15:56.155 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "complete" 2020-11-09T18:15:56.155 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:56.156 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:56.156 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:15:56.156 INFO:teuthology.orchestra.run.smithi078.stdout:] 2020-11-09T18:15:56.157 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:15:56.158 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:15:56.159 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-09T18:15:56.193 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:15:56.194 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:15:56.506 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:15:56.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:15:56.507 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-09T18:15:56.538 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:15:56.539 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:15:56.979 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:15:56.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:15:56.980 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-09T18:15:57.018 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:15:57.019 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:15:57.288 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:15:57.290 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:15:57.291 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-09T18:15:57.328 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:15:57.329 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:15:57.728 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:57.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:57.729 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:15:57.775 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:15:57.775 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:15:58.052 INFO:teuthology.orchestra.run.smithi051.stdout:[] 2020-11-09T18:15:58.054 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:58.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:58.056 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:15:58.095 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:15:58.095 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:15:58.360 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2020-11-09T18:15:58.362 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:58.363 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:58.363 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:15:58.404 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:15:58.405 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:15:58.679 INFO:teuthology.orchestra.run.smithi051.stdout:[] 2020-11-09T18:15:58.682 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:58.683 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:58.684 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:15:58.722 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:15:58.723 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:15:58.980 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2020-11-09T18:15:58.981 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'bbf9f070-21b9-4940-95e9-d031ba1c647f', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:58.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'bbf9f070-21b9-4940-95e9-d031ba1c647f', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:58.982 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period bbf9f070-21b9-4940-95e9-d031ba1c647f --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:15:59.022 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:15:59.022 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:15:59.289 INFO:teuthology.orchestra.run.smithi051.stdout:[] 2020-11-09T18:15:59.291 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'bbf9f070-21b9-4940-95e9-d031ba1c647f', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:59.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'bbf9f070-21b9-4940-95e9-d031ba1c647f', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:59.292 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period bbf9f070-21b9-4940-95e9-d031ba1c647f --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:15:59.332 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:15:59.332 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:15:59.589 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2020-11-09T18:15:59.590 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'bbf9f070-21b9-4940-95e9-d031ba1c647f', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:59.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'bbf9f070-21b9-4940-95e9-d031ba1c647f', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:59.591 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period bbf9f070-21b9-4940-95e9-d031ba1c647f --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:15:59.629 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:15:59.630 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:15:59.890 INFO:teuthology.orchestra.run.smithi051.stdout:[] 2020-11-09T18:15:59.893 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'bbf9f070-21b9-4940-95e9-d031ba1c647f', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:59.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'bbf9f070-21b9-4940-95e9-d031ba1c647f', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:15:59.894 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period bbf9f070-21b9-4940-95e9-d031ba1c647f --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:15:59.934 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:15:59.935 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:16:00.208 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2020-11-09T18:16:00.210 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'f6258aba-300a-4d5d-ad93-a7c198a0e0a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:16:00.210 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'f6258aba-300a-4d5d-ad93-a7c198a0e0a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:16:00.211 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period f6258aba-300a-4d5d-ad93-a7c198a0e0a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:16:00.248 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:16:00.249 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:16:00.537 INFO:teuthology.orchestra.run.smithi051.stdout:[] 2020-11-09T18:16:00.539 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'f6258aba-300a-4d5d-ad93-a7c198a0e0a1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:16:00.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'f6258aba-300a-4d5d-ad93-a7c198a0e0a1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:16:00.540 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period f6258aba-300a-4d5d-ad93-a7c198a0e0a1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:16:00.573 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:16:00.573 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:16:00.822 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2020-11-09T18:16:00.824 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'f6258aba-300a-4d5d-ad93-a7c198a0e0a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:16:00.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'f6258aba-300a-4d5d-ad93-a7c198a0e0a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:16:00.825 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period f6258aba-300a-4d5d-ad93-a7c198a0e0a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:16:00.865 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:16:00.865 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:16:01.131 INFO:teuthology.orchestra.run.smithi051.stdout:[] 2020-11-09T18:16:01.133 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'f6258aba-300a-4d5d-ad93-a7c198a0e0a1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:16:01.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'f6258aba-300a-4d5d-ad93-a7c198a0e0a1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:16:01.135 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period f6258aba-300a-4d5d-ad93-a7c198a0e0a1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:16:01.174 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:16:01.175 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:16:01.429 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2020-11-09T18:16:01.432 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multi_period_incremental_sync ... ok 2020-11-09T18:16:01.432 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ymzeui-46 2020-11-09T18:16:02.381 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ymzeui-47 2020-11-09T18:16:02.983 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ymzeui-48 2020-11-09T18:16:03.517 INFO:tasks.rgw.c1.client.0.smithi051.stdout:Parse error at line 1: 2020-11-09T18:16:05.030 INFO:tasks.rgw.c1.client.0.smithi051.stdout:syntax error 2020-11-09T18:16:05.031 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_datalog_autotrim ... ERROR 2020-11-09T18:16:05.032 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:16:05.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:16:05.033 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-09T18:16:05.075 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:16:05.075 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:16:05.104 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:16:05.100+0000 7fb5f36cd0c0 0 failed reading obj info from .rgw.root:zone_info.11f516a7-3288-49a6-9a89-b45ac1dae91a: (2) No such file or directory 2020-11-09T18:16:05.105 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:16:05.100+0000 7fb5f36cd0c0 0 WARNING: could not read zone params for zone id=11f516a7-3288-49a6-9a89-b45ac1dae91a name=test-zone1 2020-11-09T18:16:05.109 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:16:05.108+0000 7fb5f36cd0c0 0 failed reading obj info from .rgw.root:zone_info.a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6: (2) No such file or directory 2020-11-09T18:16:05.110 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:16:05.108+0000 7fb5f36cd0c0 0 WARNING: could not read zone params for zone id=a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6 name=test-zone3 2020-11-09T18:16:05.121 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:16:05.121 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc", 2020-11-09T18:16:05.121 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "test-zone2", 2020-11-09T18:16:05.121 INFO:teuthology.orchestra.run.smithi078.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-11-09T18:16:05.122 INFO:teuthology.orchestra.run.smithi078.stdout: "control_pool": "test-zone2.rgw.control", 2020-11-09T18:16:05.122 INFO:teuthology.orchestra.run.smithi078.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-11-09T18:16:05.122 INFO:teuthology.orchestra.run.smithi078.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-11-09T18:16:05.122 INFO:teuthology.orchestra.run.smithi078.stdout: "log_pool": "test-zone2.rgw.log", 2020-11-09T18:16:05.122 INFO:teuthology.orchestra.run.smithi078.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-11-09T18:16:05.122 INFO:teuthology.orchestra.run.smithi078.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-11-09T18:16:05.123 INFO:teuthology.orchestra.run.smithi078.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-11-09T18:16:05.123 INFO:teuthology.orchestra.run.smithi078.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-11-09T18:16:05.123 INFO:teuthology.orchestra.run.smithi078.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-11-09T18:16:05.123 INFO:teuthology.orchestra.run.smithi078.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-11-09T18:16:05.124 INFO:teuthology.orchestra.run.smithi078.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-11-09T18:16:05.124 INFO:teuthology.orchestra.run.smithi078.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-11-09T18:16:05.125 INFO:teuthology.orchestra.run.smithi078.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-11-09T18:16:05.125 INFO:teuthology.orchestra.run.smithi078.stdout: "system_key": { 2020-11-09T18:16:05.125 INFO:teuthology.orchestra.run.smithi078.stdout: "access_key": "ZAW7MMQ493TT1LMM", 2020-11-09T18:16:05.125 INFO:teuthology.orchestra.run.smithi078.stdout: "secret_key": "HQHAA9hTQSSDkU0yH5qgAQS7JzFsECDH" 2020-11-09T18:16:05.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:16:05.125 INFO:teuthology.orchestra.run.smithi078.stdout: "placement_pools": [ 2020-11-09T18:16:05.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:16:05.126 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "default-placement", 2020-11-09T18:16:05.126 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:16:05.126 INFO:teuthology.orchestra.run.smithi078.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-11-09T18:16:05.126 INFO:teuthology.orchestra.run.smithi078.stdout: "storage_classes": { 2020-11-09T18:16:05.127 INFO:teuthology.orchestra.run.smithi078.stdout: "STANDARD": { 2020-11-09T18:16:05.127 INFO:teuthology.orchestra.run.smithi078.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2020-11-09T18:16:05.127 INFO:teuthology.orchestra.run.smithi078.stdout: "compression_type": "random" 2020-11-09T18:16:05.127 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:16:05.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:16:05.127 INFO:teuthology.orchestra.run.smithi078.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-11-09T18:16:05.127 INFO:teuthology.orchestra.run.smithi078.stdout: "index_type": 0 2020-11-09T18:16:05.128 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:16:05.128 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:16:05.128 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:16:05.128 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_id": "0de9c5fb-e243-4abd-b414-59ffbc2cb575", 2020-11-09T18:16:05.128 INFO:teuthology.orchestra.run.smithi078.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2020-11-09T18:16:05.128 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:16:05.129 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:16:05.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:16:05.130 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-09T18:16:05.213 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:16:05.214 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:16:05.479 INFO:teuthology.orchestra.run.smithi078.stderr:Sending period to new master zone 11f516a7-3288-49a6-9a89-b45ac1dae91a 2020-11-09T18:16:05.551 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:16:05.552 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:16:05.552 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 2, 2020-11-09T18:16:05.552 INFO:teuthology.orchestra.run.smithi078.stdout: "predecessor_uuid": "bbf9f070-21b9-4940-95e9-d031ba1c647f", 2020-11-09T18:16:05.552 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": [ 2020-11-09T18:16:05.552 INFO:teuthology.orchestra.run.smithi078.stdout: "1_1604945680.637050_159.1", 2020-11-09T18:16:05.552 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2020-11-09T18:16:05.552 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2020-11-09T18:16:05.553 INFO:teuthology.orchestra.run.smithi078.stdout: "" 2020-11-09T18:16:05.553 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:16:05.553 INFO:teuthology.orchestra.run.smithi078.stdout: "period_map": { 2020-11-09T18:16:05.553 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:16:05.553 INFO:teuthology.orchestra.run.smithi078.stdout: "zonegroups": [ 2020-11-09T18:16:05.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:16:05.553 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "test-zonegroup", 2020-11-09T18:16:05.554 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "test-zonegroup", 2020-11-09T18:16:05.554 INFO:teuthology.orchestra.run.smithi078.stdout: "api_name": "", 2020-11-09T18:16:05.554 INFO:teuthology.orchestra.run.smithi078.stdout: "is_master": "true", 2020-11-09T18:16:05.554 INFO:teuthology.orchestra.run.smithi078.stdout: "endpoints": [ 2020-11-09T18:16:05.554 INFO:teuthology.orchestra.run.smithi078.stdout: "http://smithi051.front.sepia.ceph.com:8000" 2020-11-09T18:16:05.554 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:16:05.554 INFO:teuthology.orchestra.run.smithi078.stdout: "hostnames": [], 2020-11-09T18:16:05.555 INFO:teuthology.orchestra.run.smithi078.stdout: "hostnames_s3website": [], 2020-11-09T18:16:05.555 INFO:teuthology.orchestra.run.smithi078.stdout: "master_zone": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:16:05.555 INFO:teuthology.orchestra.run.smithi078.stdout: "zones": [ 2020-11-09T18:16:05.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:16:05.555 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:16:05.555 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "test-zone1", 2020-11-09T18:16:05.556 INFO:teuthology.orchestra.run.smithi078.stdout: "endpoints": [ 2020-11-09T18:16:05.556 INFO:teuthology.orchestra.run.smithi078.stdout: "http://smithi051.front.sepia.ceph.com:8000" 2020-11-09T18:16:05.556 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:16:05.556 INFO:teuthology.orchestra.run.smithi078.stdout: "log_meta": "false", 2020-11-09T18:16:05.556 INFO:teuthology.orchestra.run.smithi078.stdout: "log_data": "true", 2020-11-09T18:16:05.556 INFO:teuthology.orchestra.run.smithi078.stdout: "bucket_index_max_shards": 11, 2020-11-09T18:16:05.556 INFO:teuthology.orchestra.run.smithi078.stdout: "read_only": "false", 2020-11-09T18:16:05.557 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_type": "", 2020-11-09T18:16:05.557 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_from_all": "true", 2020-11-09T18:16:05.557 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_from": [], 2020-11-09T18:16:05.557 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect_zone": "" 2020-11-09T18:16:05.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:16:05.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:16:05.557 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "4600c141-233a-40b8-a0db-e3e3a34df6c3", 2020-11-09T18:16:05.558 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "test-zone4", 2020-11-09T18:16:05.558 INFO:teuthology.orchestra.run.smithi078.stdout: "endpoints": [ 2020-11-09T18:16:05.558 INFO:teuthology.orchestra.run.smithi078.stdout: "http://smithi078.front.sepia.ceph.com:8001" 2020-11-09T18:16:05.558 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:16:05.558 INFO:teuthology.orchestra.run.smithi078.stdout: "log_meta": "false", 2020-11-09T18:16:05.558 INFO:teuthology.orchestra.run.smithi078.stdout: "log_data": "true", 2020-11-09T18:16:05.558 INFO:teuthology.orchestra.run.smithi078.stdout: "bucket_index_max_shards": 11, 2020-11-09T18:16:05.559 INFO:teuthology.orchestra.run.smithi078.stdout: "read_only": "false", 2020-11-09T18:16:05.559 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_type": "pubsub", 2020-11-09T18:16:05.559 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_from_all": "false", 2020-11-09T18:16:05.559 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_from": [ 2020-11-09T18:16:05.559 INFO:teuthology.orchestra.run.smithi078.stdout: "test-zone1" 2020-11-09T18:16:05.559 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:16:05.559 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect_zone": "" 2020-11-09T18:16:05.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:16:05.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:16:05.560 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc", 2020-11-09T18:16:05.560 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "test-zone2", 2020-11-09T18:16:05.560 INFO:teuthology.orchestra.run.smithi078.stdout: "endpoints": [ 2020-11-09T18:16:05.560 INFO:teuthology.orchestra.run.smithi078.stdout: "http://smithi078.front.sepia.ceph.com:8000" 2020-11-09T18:16:05.560 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:16:05.561 INFO:teuthology.orchestra.run.smithi078.stdout: "log_meta": "false", 2020-11-09T18:16:05.561 INFO:teuthology.orchestra.run.smithi078.stdout: "log_data": "true", 2020-11-09T18:16:05.561 INFO:teuthology.orchestra.run.smithi078.stdout: "bucket_index_max_shards": 11, 2020-11-09T18:16:05.561 INFO:teuthology.orchestra.run.smithi078.stdout: "read_only": "false", 2020-11-09T18:16:05.561 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_type": "", 2020-11-09T18:16:05.561 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_from_all": "false", 2020-11-09T18:16:05.561 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_from": [], 2020-11-09T18:16:05.561 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect_zone": "" 2020-11-09T18:16:05.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:16:05.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:16:05.562 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6", 2020-11-09T18:16:05.562 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "test-zone3", 2020-11-09T18:16:05.562 INFO:teuthology.orchestra.run.smithi078.stdout: "endpoints": [ 2020-11-09T18:16:05.562 INFO:teuthology.orchestra.run.smithi078.stdout: "http://smithi051.front.sepia.ceph.com:8001" 2020-11-09T18:16:05.563 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:16:05.563 INFO:teuthology.orchestra.run.smithi078.stdout: "log_meta": "false", 2020-11-09T18:16:05.563 INFO:teuthology.orchestra.run.smithi078.stdout: "log_data": "true", 2020-11-09T18:16:05.563 INFO:teuthology.orchestra.run.smithi078.stdout: "bucket_index_max_shards": 11, 2020-11-09T18:16:05.563 INFO:teuthology.orchestra.run.smithi078.stdout: "read_only": "false", 2020-11-09T18:16:05.563 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_type": "", 2020-11-09T18:16:05.563 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_from_all": "true", 2020-11-09T18:16:05.563 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_from": [], 2020-11-09T18:16:05.564 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect_zone": "" 2020-11-09T18:16:05.564 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:16:05.564 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:16:05.564 INFO:teuthology.orchestra.run.smithi078.stdout: "placement_targets": [ 2020-11-09T18:16:05.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:16:05.564 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "default-placement", 2020-11-09T18:16:05.564 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2020-11-09T18:16:05.565 INFO:teuthology.orchestra.run.smithi078.stdout: "storage_classes": [ 2020-11-09T18:16:05.565 INFO:teuthology.orchestra.run.smithi078.stdout: "STANDARD" 2020-11-09T18:16:05.565 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:16:05.565 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:16:05.565 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:16:05.565 INFO:teuthology.orchestra.run.smithi078.stdout: "default_placement": "default-placement", 2020-11-09T18:16:05.565 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_id": "0de9c5fb-e243-4abd-b414-59ffbc2cb575", 2020-11-09T18:16:05.566 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_policy": { 2020-11-09T18:16:05.566 INFO:teuthology.orchestra.run.smithi078.stdout: "groups": [] 2020-11-09T18:16:05.566 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:16:05.566 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:16:05.566 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:16:05.581 INFO:teuthology.orchestra.run.smithi078.stdout: "short_zone_ids": [ 2020-11-09T18:16:05.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:16:05.582 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:16:05.582 INFO:teuthology.orchestra.run.smithi078.stdout: "val": 2019612975 2020-11-09T18:16:05.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:16:05.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:16:05.582 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "4600c141-233a-40b8-a0db-e3e3a34df6c3", 2020-11-09T18:16:05.582 INFO:teuthology.orchestra.run.smithi078.stdout: "val": 1720005997 2020-11-09T18:16:05.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:16:05.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:16:05.583 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc", 2020-11-09T18:16:05.583 INFO:teuthology.orchestra.run.smithi078.stdout: "val": 2281443694 2020-11-09T18:16:05.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:16:05.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:16:05.583 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6", 2020-11-09T18:16:05.584 INFO:teuthology.orchestra.run.smithi078.stdout: "val": 648291153 2020-11-09T18:16:05.584 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:16:05.584 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:16:05.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:16:05.584 INFO:teuthology.orchestra.run.smithi078.stdout: "master_zonegroup": "test-zonegroup", 2020-11-09T18:16:05.584 INFO:teuthology.orchestra.run.smithi078.stdout: "master_zone": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:16:05.584 INFO:teuthology.orchestra.run.smithi078.stdout: "period_config": { 2020-11-09T18:16:05.585 INFO:teuthology.orchestra.run.smithi078.stdout: "bucket_quota": { 2020-11-09T18:16:05.585 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": false, 2020-11-09T18:16:05.585 INFO:teuthology.orchestra.run.smithi078.stdout: "check_on_raw": false, 2020-11-09T18:16:05.585 INFO:teuthology.orchestra.run.smithi078.stdout: "max_size": -1, 2020-11-09T18:16:05.585 INFO:teuthology.orchestra.run.smithi078.stdout: "max_size_kb": 0, 2020-11-09T18:16:05.585 INFO:teuthology.orchestra.run.smithi078.stdout: "max_objects": -1 2020-11-09T18:16:05.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:16:05.585 INFO:teuthology.orchestra.run.smithi078.stdout: "user_quota": { 2020-11-09T18:16:05.586 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": false, 2020-11-09T18:16:05.586 INFO:teuthology.orchestra.run.smithi078.stdout: "check_on_raw": false, 2020-11-09T18:16:05.586 INFO:teuthology.orchestra.run.smithi078.stdout: "max_size": -1, 2020-11-09T18:16:05.586 INFO:teuthology.orchestra.run.smithi078.stdout: "max_size_kb": 0, 2020-11-09T18:16:05.586 INFO:teuthology.orchestra.run.smithi078.stdout: "max_objects": -1 2020-11-09T18:16:05.586 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:16:05.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:16:05.587 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_id": "0de9c5fb-e243-4abd-b414-59ffbc2cb575", 2020-11-09T18:16:05.587 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_name": "test-realm", 2020-11-09T18:16:05.587 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:16:05.587 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:16:05.588 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to false 2020-11-09T18:17:05.589 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ymzeui-49 2020-11-09T18:17:05.640 INFO:tasks.rgw.c1.client.0.smithi051.stdout:Parse error at line 1: 2020-11-09T18:17:05.640 INFO:tasks.rgw.c1.client.0.smithi051.stdout:syntax error 2020-11-09T18:17:05.683 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multi_zone_redirect ... ERROR 2020-11-09T18:17:05.684 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:17:05.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:17:05.686 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:17:05.727 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:17:05.727 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:17:05.946 INFO:teuthology.orchestra.run.smithi051.stderr:No --period given, using current period=f6258aba-300a-4d5d-ad93-a7c198a0e0a1 2020-11-09T18:17:05.981 INFO:teuthology.orchestra.run.smithi051.stdout:[ 2020-11-09T18:17:05.981 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:17:05.981 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945825.625804_254.1", 2020-11-09T18:17:05.982 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "2020-11-09T18:17:05.625804Z" 2020-11-09T18:17:05.982 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:17:05.982 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:17:05.982 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:17:05.982 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:17:05.982 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:17:05.983 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:17:05.983 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:17:05.983 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:17:05.983 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:17:05.983 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:17:05.983 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:17:05.983 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:17:05.984 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:17:05.984 INFO:teuthology.orchestra.run.smithi051.stdout:] 2020-11-09T18:17:05.985 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-09T18:17:05.986 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:17:05.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:17:05.988 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:17:06.028 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:17:06.028 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:17:06.274 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:17:06.275 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:17:06.275 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:17:06.275 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:17:06.275 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:17:06.275 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:17:06.275 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:17:06.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:06.276 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:17:06.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:17:06.276 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:17:06.276 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:17:06.276 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:17:06.276 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945763.415139_233.1", 2020-11-09T18:17:06.277 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:17:06.277 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:17:06.277 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:17:06.277 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:16:03.415139Z", 2020-11-09T18:17:06.277 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:17:06.277 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:06.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:06.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:17:06.278 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:17:06.278 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:17:06.278 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:17:06.278 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:17:06.278 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:17:06.278 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:17:06.279 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:17:06.279 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:17:06.279 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:17:06.279 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:06.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:06.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:17:06.279 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:17:06.280 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:17:06.280 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:17:06.280 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:17:06.280 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:17:06.280 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:17:06.280 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:17:06.280 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:17:06.280 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:17:06.281 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:06.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:06.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:17:06.281 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:17:06.281 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:17:06.281 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:17:06.281 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:17:06.282 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:17:06.282 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:17:06.282 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:17:06.282 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:17:06.282 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:17:06.282 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:06.282 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:06.283 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:17:06.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:06.283 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:17:06.283 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:17:06.283 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:17:06.283 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:06.283 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:17:06.284 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945825.625804_254.1 target=1_1604945763.415139_233.1 2020-11-09T18:17:11.285 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:17:11.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:17:11.287 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:17:11.329 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:17:11.329 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:17:11.583 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:17:11.583 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:17:11.583 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:17:11.583 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:17:11.583 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:17:11.584 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:17:11.584 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:17:11.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:11.584 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:17:11.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:17:11.584 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:17:11.584 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:17:11.585 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:17:11.585 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945763.415139_233.1", 2020-11-09T18:17:11.585 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:17:11.585 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:17:11.585 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:17:11.585 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:16:03.415139Z", 2020-11-09T18:17:11.585 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:17:11.586 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:11.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:11.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:17:11.586 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:17:11.586 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:17:11.586 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:17:11.586 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:17:11.587 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:17:11.587 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:17:11.587 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:17:11.587 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:17:11.587 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:17:11.587 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:11.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:11.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:17:11.588 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:17:11.588 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:17:11.588 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:17:11.588 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:17:11.588 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:17:11.588 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:17:11.588 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:17:11.589 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:17:11.589 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:17:11.589 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:11.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:11.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:17:11.589 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:17:11.589 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:17:11.590 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:17:11.590 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:17:11.590 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:17:11.590 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:17:11.590 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:17:11.590 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:17:11.590 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:17:11.590 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:11.591 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:11.591 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:17:11.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:11.591 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:17:11.591 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:17:11.591 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:17:11.591 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:11.592 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:17:11.592 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945825.625804_254.1 target=1_1604945763.415139_233.1 2020-11-09T18:17:16.594 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:17:16.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:17:16.596 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:17:16.638 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:17:16.639 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:17:16.884 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:17:16.884 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:17:16.884 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:17:16.884 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:17:16.884 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:17:16.885 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:17:16.885 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:17:16.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:16.885 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:17:16.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:17:16.885 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:17:16.886 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:17:16.886 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:17:16.886 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945763.415139_233.1", 2020-11-09T18:17:16.886 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:17:16.886 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:17:16.886 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:17:16.886 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:16:03.415139Z", 2020-11-09T18:17:16.887 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:17:16.887 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:16.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:16.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:17:16.887 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:17:16.887 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:17:16.887 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:17:16.887 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:17:16.888 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:17:16.888 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:17:16.888 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:17:16.888 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:17:16.888 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:17:16.888 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:16.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:16.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:17:16.889 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:17:16.889 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:17:16.889 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:17:16.889 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:17:16.889 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:17:16.889 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:17:16.889 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:17:16.890 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:17:16.890 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:17:16.890 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:16.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:16.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:17:16.890 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:17:16.890 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:17:16.891 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:17:16.891 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:17:16.891 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:17:16.891 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:17:16.891 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:17:16.891 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:17:16.891 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:17:16.892 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:16.892 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:16.892 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:17:16.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:16.892 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:17:16.892 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:17:16.892 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:17:16.892 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:16.893 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:17:16.893 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945825.625804_254.1 target=1_1604945763.415139_233.1 2020-11-09T18:17:21.894 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:17:21.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:17:21.896 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:17:21.936 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:17:21.936 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:17:22.194 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:17:22.195 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:17:22.195 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:17:22.195 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:17:22.195 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:17:22.195 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:17:22.195 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:17:22.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:22.196 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:17:22.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:17:22.196 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:17:22.196 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:17:22.196 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:17:22.196 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945763.415139_233.1", 2020-11-09T18:17:22.197 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:17:22.197 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:17:22.197 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:17:22.197 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:16:03.415139Z", 2020-11-09T18:17:22.197 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:17:22.197 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:22.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:22.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:17:22.198 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:17:22.198 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:17:22.198 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:17:22.198 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:17:22.198 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:17:22.198 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:17:22.199 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:17:22.199 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:17:22.199 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:17:22.199 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:22.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:22.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:17:22.199 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:17:22.199 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:17:22.200 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:17:22.200 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:17:22.200 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:17:22.200 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:17:22.200 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:17:22.200 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:17:22.200 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:17:22.201 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:22.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:22.201 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:17:22.201 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:17:22.201 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:17:22.201 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:17:22.201 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:17:22.201 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:17:22.202 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:17:22.202 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:17:22.202 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:17:22.202 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:17:22.202 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:22.202 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:22.202 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:17:22.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:22.203 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:17:22.203 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:17:22.203 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:17:22.203 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:22.203 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:17:22.204 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945825.625804_254.1 target=1_1604945763.415139_233.1 2020-11-09T18:17:27.206 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:17:27.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:17:27.207 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:17:27.248 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:17:27.249 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:17:27.501 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:17:27.501 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:17:27.502 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:17:27.502 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:17:27.502 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:17:27.502 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:17:27.502 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:17:27.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:27.502 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:17:27.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:17:27.503 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:17:27.503 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:17:27.503 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:17:27.503 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945763.415139_233.1", 2020-11-09T18:17:27.503 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:17:27.503 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:17:27.504 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:17:27.504 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:16:03.415139Z", 2020-11-09T18:17:27.504 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:17:27.504 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:27.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:27.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:17:27.504 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:17:27.505 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:17:27.505 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:17:27.505 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:17:27.505 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:17:27.505 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:17:27.505 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:17:27.505 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:17:27.506 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:17:27.506 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:27.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:27.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:17:27.506 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:17:27.506 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:17:27.506 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:17:27.507 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:17:27.507 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:17:27.507 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:17:27.507 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:17:27.507 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:17:27.507 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:17:27.507 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:27.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:27.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:17:27.508 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:17:27.508 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:17:27.508 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:17:27.508 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:17:27.508 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:17:27.508 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:17:27.509 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:17:27.509 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:17:27.509 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:17:27.509 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:27.509 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:27.509 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:17:27.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:27.510 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:17:27.510 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:17:27.510 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:17:27.510 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:27.510 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:17:27.511 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945825.625804_254.1 target=1_1604945763.415139_233.1 2020-11-09T18:17:32.512 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:17:32.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:17:32.514 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:17:32.549 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:17:32.550 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:17:32.795 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:17:32.795 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:17:32.795 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:17:32.795 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:17:32.796 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:17:32.796 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:17:32.796 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:17:32.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:32.796 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:17:32.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:17:32.797 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:17:32.797 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:17:32.797 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:17:32.797 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945763.415139_233.1", 2020-11-09T18:17:32.797 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:17:32.797 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:17:32.797 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:17:32.798 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:16:03.415139Z", 2020-11-09T18:17:32.798 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:17:32.798 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:32.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:32.798 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:17:32.798 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:17:32.798 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:17:32.799 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:17:32.799 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:17:32.799 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:17:32.799 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:17:32.799 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:17:32.799 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:17:32.799 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:17:32.799 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:32.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:32.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:17:32.800 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:17:32.800 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:17:32.800 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:17:32.800 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:17:32.800 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:17:32.801 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:17:32.801 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:17:32.801 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:17:32.801 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:17:32.801 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:32.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:32.801 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:17:32.802 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:17:32.802 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:17:32.802 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:17:32.802 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:17:32.802 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:17:32.802 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:17:32.802 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:17:32.803 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:17:32.803 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:17:32.803 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:32.803 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:32.803 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:17:32.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:32.803 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:17:32.804 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:17:32.804 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:17:32.804 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:32.804 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:17:32.805 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945825.625804_254.1 target=1_1604945763.415139_233.1 2020-11-09T18:17:37.806 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:17:37.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:17:37.807 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:17:37.849 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:17:37.850 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:17:38.107 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:17:38.107 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:17:38.107 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:17:38.108 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:17:38.108 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:17:38.108 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:17:38.108 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:17:38.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:38.108 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:17:38.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:17:38.109 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:17:38.109 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:17:38.109 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:17:38.109 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945763.415139_233.1", 2020-11-09T18:17:38.109 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:17:38.109 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:17:38.109 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:17:38.110 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:16:03.415139Z", 2020-11-09T18:17:38.110 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:17:38.110 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:38.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:38.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:17:38.110 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:17:38.110 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:17:38.111 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:17:38.111 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:17:38.111 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:17:38.111 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:17:38.111 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:17:38.111 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:17:38.111 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:17:38.112 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:38.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:38.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:17:38.112 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:17:38.112 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:17:38.112 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:17:38.112 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:17:38.112 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:17:38.113 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:17:38.113 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:17:38.113 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:17:38.113 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:17:38.113 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:38.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:38.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:17:38.114 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:17:38.114 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:17:38.114 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:17:38.114 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:17:38.114 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:17:38.114 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:17:38.114 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:17:38.115 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:17:38.116 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:17:38.116 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:38.117 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:38.117 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:17:38.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:38.117 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:17:38.117 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:17:38.117 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:17:38.118 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:38.118 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:17:38.119 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945825.625804_254.1 target=1_1604945763.415139_233.1 2020-11-09T18:17:43.120 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:17:43.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:17:43.121 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:17:43.163 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:17:43.163 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:17:43.417 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:17:43.418 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:17:43.418 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:17:43.418 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:17:43.418 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:17:43.418 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:17:43.419 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:17:43.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:43.419 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:17:43.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:17:43.419 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:17:43.419 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:17:43.420 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:17:43.420 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945763.415139_233.1", 2020-11-09T18:17:43.420 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:17:43.420 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:17:43.421 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:17:43.421 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:16:03.415139Z", 2020-11-09T18:17:43.421 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:17:43.421 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:43.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:43.422 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:17:43.422 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:17:43.422 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:17:43.422 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:17:43.422 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:17:43.423 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:17:43.423 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:17:43.423 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:17:43.423 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:17:43.423 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:17:43.423 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:43.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:43.424 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:17:43.424 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:17:43.424 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:17:43.424 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:17:43.424 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:17:43.425 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:17:43.425 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:17:43.425 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:17:43.425 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:17:43.425 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:17:43.426 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:43.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:43.426 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:17:43.426 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:17:43.426 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:17:43.427 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:17:43.427 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:17:43.427 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:17:43.427 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:17:43.427 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:17:43.427 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:17:43.427 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:17:43.428 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:43.428 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:43.428 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:17:43.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:43.428 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:17:43.428 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:17:43.429 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:17:43.429 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:43.429 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:17:43.430 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945825.625804_254.1 target=1_1604945763.415139_233.1 2020-11-09T18:17:48.431 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:17:48.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:17:48.432 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:17:48.472 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:17:48.472 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:17:48.701 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:17:48.702 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:17:48.702 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:17:48.702 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:17:48.702 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:17:48.702 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:17:48.703 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:17:48.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:48.703 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:17:48.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:17:48.703 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:17:48.703 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:17:48.703 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:17:48.704 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945763.415139_233.1", 2020-11-09T18:17:48.704 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:17:48.704 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:17:48.704 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:17:48.704 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:16:03.415139Z", 2020-11-09T18:17:48.704 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:17:48.706 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:48.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:48.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:17:48.707 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:17:48.707 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:17:48.707 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:17:48.707 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:17:48.707 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:17:48.708 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:17:48.708 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:17:48.708 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:17:48.708 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:17:48.708 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:48.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:48.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:17:48.709 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:17:48.709 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:17:48.709 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:17:48.709 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:17:48.710 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:17:48.710 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:17:48.710 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:17:48.711 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:17:48.711 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:17:48.711 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:48.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:48.712 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:17:48.712 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:17:48.712 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:17:48.712 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:17:48.713 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:17:48.713 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:17:48.713 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:17:48.713 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:17:48.714 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:17:48.714 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:17:48.714 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:48.714 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:48.715 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:17:48.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:48.715 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:17:48.715 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:17:48.716 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:17:48.716 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:48.716 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:17:48.718 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945825.625804_254.1 target=1_1604945763.415139_233.1 2020-11-09T18:17:53.719 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:17:53.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:17:53.721 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:17:53.764 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:17:53.765 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:17:54.013 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:17:54.013 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:17:54.014 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:17:54.014 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:17:54.014 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:17:54.014 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:17:54.014 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:17:54.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:54.015 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:17:54.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:17:54.015 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:17:54.015 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:17:54.015 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:17:54.015 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945763.415139_233.1", 2020-11-09T18:17:54.015 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:17:54.016 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:17:54.016 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:17:54.016 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:16:03.415139Z", 2020-11-09T18:17:54.016 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:17:54.016 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:54.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:54.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:17:54.017 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:17:54.017 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:17:54.017 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:17:54.017 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:17:54.017 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:17:54.017 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:17:54.017 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:17:54.017 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:17:54.018 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:17:54.018 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:54.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:54.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:17:54.018 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:17:54.018 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:17:54.018 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:17:54.019 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:17:54.019 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:17:54.019 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:17:54.019 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:17:54.019 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:17:54.019 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:17:54.019 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:54.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:54.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:17:54.020 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:17:54.020 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:17:54.020 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:17:54.020 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:17:54.020 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:17:54.021 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:17:54.021 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:17:54.021 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:17:54.021 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:17:54.021 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:54.021 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:54.021 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:17:54.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:54.022 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:17:54.022 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:17:54.022 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:17:54.022 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:54.022 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:17:54.024 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945825.625804_254.1 target=1_1604945763.415139_233.1 2020-11-09T18:17:59.025 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:17:59.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:17:59.028 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:17:59.068 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:17:59.069 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:17:59.317 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:17:59.317 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:17:59.317 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:17:59.318 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:17:59.318 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:17:59.318 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:17:59.318 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:17:59.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:59.318 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:17:59.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:17:59.319 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:17:59.319 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:17:59.319 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:17:59.319 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945763.415139_233.1", 2020-11-09T18:17:59.319 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:17:59.320 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:17:59.320 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:17:59.320 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:16:03.415139Z", 2020-11-09T18:17:59.320 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:17:59.320 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:59.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:59.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:17:59.321 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:17:59.321 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:17:59.321 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:17:59.321 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:17:59.321 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:17:59.322 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:17:59.322 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:17:59.322 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:17:59.322 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:17:59.322 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:59.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:59.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:17:59.323 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:17:59.323 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:17:59.323 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:17:59.323 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:17:59.323 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:17:59.324 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:17:59.324 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:17:59.324 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:17:59.324 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:17:59.324 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:59.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:59.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:17:59.325 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:17:59.325 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:17:59.325 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:17:59.325 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:17:59.325 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:17:59.326 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:17:59.326 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:17:59.326 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:17:59.326 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:17:59.326 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:59.326 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:59.327 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:17:59.327 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:17:59.327 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:17:59.327 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:17:59.327 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:17:59.327 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:17:59.328 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:17:59.329 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945825.625804_254.1 target=1_1604945763.415139_233.1 2020-11-09T18:18:04.330 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:18:04.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:18:04.331 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:18:04.373 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:18:04.373 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:18:04.612 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:18:04.612 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:18:04.612 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:18:04.612 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:18:04.613 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:18:04.613 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:18:04.613 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:18:04.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:04.613 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:18:04.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:04.613 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:18:04.614 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:18:04.614 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:18:04.614 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945763.415139_233.1", 2020-11-09T18:18:04.614 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:18:04.614 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:18:04.614 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:18:04.614 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:16:03.415139Z", 2020-11-09T18:18:04.615 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:18:04.615 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:04.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:04.615 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:04.615 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:18:04.615 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:18:04.615 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:18:04.616 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:18:04.616 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:18:04.616 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:18:04.616 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:18:04.616 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:18:04.617 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:18:04.617 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:04.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:04.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:04.618 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:18:04.618 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:18:04.618 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:18:04.618 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:18:04.618 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:18:04.618 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:18:04.618 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:18:04.619 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:18:04.619 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:18:04.619 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:04.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:04.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:04.619 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:18:04.619 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:18:04.619 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:18:04.620 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:18:04.620 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:18:04.620 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:18:04.620 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:18:04.620 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:18:04.620 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:18:04.620 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:04.621 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:04.621 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:18:04.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:04.621 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:18:04.621 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:18:04.621 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:18:04.621 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:04.622 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:18:04.623 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945825.625804_254.1 target=1_1604945763.415139_233.1 2020-11-09T18:18:09.624 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:18:09.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:18:09.625 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:18:09.663 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:18:09.664 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:18:09.906 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:18:09.906 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:18:09.906 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:18:09.906 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:18:09.906 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:18:09.907 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:18:09.907 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:18:09.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:09.907 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:18:09.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:09.907 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:18:09.907 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:18:09.908 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:18:09.908 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945825.625804_254.1", 2020-11-09T18:18:09.908 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:18:09.908 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:18:09.908 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:18:09.908 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:17:05.625804Z", 2020-11-09T18:18:09.909 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:18:09.909 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:09.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:09.909 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:09.909 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:18:09.909 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:18:09.909 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:18:09.910 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:18:09.910 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:18:09.910 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:18:09.910 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:18:09.910 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:18:09.910 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:18:09.910 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:09.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:09.911 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:09.911 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:18:09.911 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:18:09.911 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:18:09.911 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:18:09.911 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:18:09.912 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:18:09.913 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:18:09.913 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:18:09.913 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:18:09.913 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:09.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:09.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:09.913 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:18:09.914 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:18:09.914 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:18:09.914 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:18:09.914 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:18:09.914 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:18:09.914 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:18:09.915 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:18:09.915 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:18:09.915 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:09.915 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:09.915 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:18:09.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:09.915 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:18:09.916 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:18:09.916 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:18:09.916 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:09.916 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:18:09.918 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-09T18:18:09.918 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-09T18:18:09.918 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:18:09.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:18:09.919 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:18:09.959 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:18:09.959 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:18:10.225 INFO:teuthology.orchestra.run.smithi051.stdout:{ 2020-11-09T18:18:10.226 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_status": { 2020-11-09T18:18:10.226 INFO:teuthology.orchestra.run.smithi051.stdout: "info": { 2020-11-09T18:18:10.226 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "sync", 2020-11-09T18:18:10.226 INFO:teuthology.orchestra.run.smithi051.stdout: "num_shards": 4, 2020-11-09T18:18:10.226 INFO:teuthology.orchestra.run.smithi051.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:18:10.227 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 4 2020-11-09T18:18:10.227 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:18:10.227 INFO:teuthology.orchestra.run.smithi051.stdout: "markers": [ 2020-11-09T18:18:10.227 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:10.227 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 0, 2020-11-09T18:18:10.227 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:18:10.227 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:18:10.228 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945825.625804_254.1", 2020-11-09T18:18:10.228 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:18:10.228 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 1, 2020-11-09T18:18:10.228 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:18:10.228 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:17:05.625804Z", 2020-11-09T18:18:10.228 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 4 2020-11-09T18:18:10.228 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:18:10.229 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:18:10.229 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:10.229 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 1, 2020-11-09T18:18:10.229 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:18:10.229 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:18:10.229 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:18:10.229 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:18:10.230 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:18:10.230 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:18:10.230 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:18:10.230 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:18:10.230 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:18:10.230 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:18:10.230 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:10.231 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 2, 2020-11-09T18:18:10.231 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:18:10.231 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:18:10.231 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:18:10.231 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:18:10.231 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:18:10.231 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:18:10.231 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:18:10.232 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:18:10.232 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:18:10.232 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:18:10.232 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:10.232 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 3, 2020-11-09T18:18:10.232 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:18:10.232 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:18:10.233 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:18:10.233 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:18:10.233 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:18:10.233 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:18:10.233 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:18:10.233 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:18:10.233 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:18:10.234 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:18:10.234 INFO:teuthology.orchestra.run.smithi051.stdout: ] 2020-11-09T18:18:10.234 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:18:10.234 INFO:teuthology.orchestra.run.smithi051.stdout: "full_sync": { 2020-11-09T18:18:10.234 INFO:teuthology.orchestra.run.smithi051.stdout: "total": 1, 2020-11-09T18:18:10.234 INFO:teuthology.orchestra.run.smithi051.stdout: "complete": 1 2020-11-09T18:18:10.234 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:18:10.234 INFO:teuthology.orchestra.run.smithi051.stdout:} 2020-11-09T18:18:10.235 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-09T18:18:10.236 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-09T18:18:10.236 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:18:10.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:18:10.237 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:18:10.276 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:18:10.276 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:18:10.541 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:18:10.542 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:18:10.542 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:18:10.542 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:18:10.542 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:18:10.542 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:18:10.543 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:18:10.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:10.543 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:18:10.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:10.543 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:18:10.543 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:18:10.543 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:18:10.544 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945825.625804_254.1", 2020-11-09T18:18:10.544 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:18:10.544 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:18:10.544 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:18:10.544 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:17:05.625804Z", 2020-11-09T18:18:10.544 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:18:10.544 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:10.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:10.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:10.545 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:18:10.545 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:18:10.545 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:18:10.545 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:18:10.545 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:18:10.546 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:18:10.546 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:18:10.546 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:18:10.546 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:18:10.546 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:10.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:10.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:10.547 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:18:10.547 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:18:10.547 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:18:10.547 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:18:10.547 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:18:10.547 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:18:10.547 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:18:10.548 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:18:10.548 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:18:10.548 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:10.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:10.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:10.548 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:18:10.548 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:18:10.549 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:18:10.549 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:18:10.549 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:18:10.549 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:18:10.550 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:18:10.550 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:18:10.550 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:18:10.550 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:10.550 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:10.550 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:18:10.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:10.551 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:18:10.551 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:18:10.551 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:18:10.551 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:10.551 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:18:10.552 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-09T18:18:10.553 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'ZAW7MMQ493TT1LMM', '--secret', 'HQHAA9hTQSSDkU0yH5qgAQS7JzFsECDH', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:18:10.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'ZAW7MMQ493TT1LMM', '--secret', 'HQHAA9hTQSSDkU0yH5qgAQS7JzFsECDH', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:18:10.554 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key ZAW7MMQ493TT1LMM --secret HQHAA9hTQSSDkU0yH5qgAQS7JzFsECDH --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-09T18:18:10.635 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:18:10.635 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:18:10.702 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:18:10.693+0000 7f839013c0c0 0 failed reading obj info from .rgw.root:zone_info.11f516a7-3288-49a6-9a89-b45ac1dae91a: (2) No such file or directory 2020-11-09T18:18:10.703 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:18:10.693+0000 7f839013c0c0 0 WARNING: could not read zone params for zone id=11f516a7-3288-49a6-9a89-b45ac1dae91a name=test-zone1 2020-11-09T18:18:10.705 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:18:10.697+0000 7f839013c0c0 0 failed reading obj info from .rgw.root:zone_info.a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6: (2) No such file or directory 2020-11-09T18:18:10.706 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:18:10.697+0000 7f839013c0c0 0 WARNING: could not read zone params for zone id=a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6 name=test-zone3 2020-11-09T18:18:10.713 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:18:10.713 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1fe6c3d0-3a82-4fac-b1e9-7bfe3c75b156", 2020-11-09T18:18:10.714 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "remove", 2020-11-09T18:18:10.714 INFO:teuthology.orchestra.run.smithi078.stdout: "domain_root": "remove.rgw.meta:root", 2020-11-09T18:18:10.714 INFO:teuthology.orchestra.run.smithi078.stdout: "control_pool": "remove.rgw.control", 2020-11-09T18:18:10.714 INFO:teuthology.orchestra.run.smithi078.stdout: "gc_pool": "remove.rgw.log:gc", 2020-11-09T18:18:10.714 INFO:teuthology.orchestra.run.smithi078.stdout: "lc_pool": "remove.rgw.log:lc", 2020-11-09T18:18:10.714 INFO:teuthology.orchestra.run.smithi078.stdout: "log_pool": "remove.rgw.log", 2020-11-09T18:18:10.715 INFO:teuthology.orchestra.run.smithi078.stdout: "intent_log_pool": "remove.rgw.log:intent", 2020-11-09T18:18:10.715 INFO:teuthology.orchestra.run.smithi078.stdout: "usage_log_pool": "remove.rgw.log:usage", 2020-11-09T18:18:10.715 INFO:teuthology.orchestra.run.smithi078.stdout: "roles_pool": "remove.rgw.meta:roles", 2020-11-09T18:18:10.715 INFO:teuthology.orchestra.run.smithi078.stdout: "reshard_pool": "remove.rgw.log:reshard", 2020-11-09T18:18:10.715 INFO:teuthology.orchestra.run.smithi078.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2020-11-09T18:18:10.715 INFO:teuthology.orchestra.run.smithi078.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2020-11-09T18:18:10.715 INFO:teuthology.orchestra.run.smithi078.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2020-11-09T18:18:10.716 INFO:teuthology.orchestra.run.smithi078.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2020-11-09T18:18:10.716 INFO:teuthology.orchestra.run.smithi078.stdout: "otp_pool": "remove.rgw.otp", 2020-11-09T18:18:10.716 INFO:teuthology.orchestra.run.smithi078.stdout: "system_key": { 2020-11-09T18:18:10.716 INFO:teuthology.orchestra.run.smithi078.stdout: "access_key": "ZAW7MMQ493TT1LMM", 2020-11-09T18:18:10.717 INFO:teuthology.orchestra.run.smithi078.stdout: "secret_key": "HQHAA9hTQSSDkU0yH5qgAQS7JzFsECDH" 2020-11-09T18:18:10.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:10.717 INFO:teuthology.orchestra.run.smithi078.stdout: "placement_pools": [ 2020-11-09T18:18:10.717 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:10.718 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "default-placement", 2020-11-09T18:18:10.718 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:18:10.718 INFO:teuthology.orchestra.run.smithi078.stdout: "index_pool": "remove.rgw.buckets.index", 2020-11-09T18:18:10.718 INFO:teuthology.orchestra.run.smithi078.stdout: "storage_classes": { 2020-11-09T18:18:10.718 INFO:teuthology.orchestra.run.smithi078.stdout: "STANDARD": { 2020-11-09T18:18:10.718 INFO:teuthology.orchestra.run.smithi078.stdout: "data_pool": "remove.rgw.buckets.data" 2020-11-09T18:18:10.718 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:10.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:10.719 INFO:teuthology.orchestra.run.smithi078.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2020-11-09T18:18:10.719 INFO:teuthology.orchestra.run.smithi078.stdout: "index_type": 0 2020-11-09T18:18:10.719 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:10.719 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:10.719 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:18:10.720 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_id": "0de9c5fb-e243-4abd-b414-59ffbc2cb575", 2020-11-09T18:18:10.721 INFO:teuthology.orchestra.run.smithi078.stdout: "notif_pool": "remove.rgw.log:notif" 2020-11-09T18:18:10.721 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:18:10.722 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:18:10.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:18:10.724 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-09T18:18:10.804 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:18:10.805 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:18:18.858 INFO:teuthology.orchestra.run.smithi078.stderr:Sending period to new master zone 11f516a7-3288-49a6-9a89-b45ac1dae91a 2020-11-09T18:18:18.938 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:18:18.938 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:18:18.938 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 3, 2020-11-09T18:18:18.939 INFO:teuthology.orchestra.run.smithi078.stdout: "predecessor_uuid": "bbf9f070-21b9-4940-95e9-d031ba1c647f", 2020-11-09T18:18:18.939 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": [ 2020-11-09T18:18:18.939 INFO:teuthology.orchestra.run.smithi078.stdout: "1_1604945680.637050_159.1", 2020-11-09T18:18:18.939 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2020-11-09T18:18:18.939 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2020-11-09T18:18:18.940 INFO:teuthology.orchestra.run.smithi078.stdout: "" 2020-11-09T18:18:18.940 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:18:18.940 INFO:teuthology.orchestra.run.smithi078.stdout: "period_map": { 2020-11-09T18:18:18.940 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:18:18.940 INFO:teuthology.orchestra.run.smithi078.stdout: "zonegroups": [ 2020-11-09T18:18:18.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:18.940 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "test-zonegroup", 2020-11-09T18:18:18.941 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "test-zonegroup", 2020-11-09T18:18:18.941 INFO:teuthology.orchestra.run.smithi078.stdout: "api_name": "", 2020-11-09T18:18:18.941 INFO:teuthology.orchestra.run.smithi078.stdout: "is_master": "true", 2020-11-09T18:18:18.941 INFO:teuthology.orchestra.run.smithi078.stdout: "endpoints": [ 2020-11-09T18:18:18.941 INFO:teuthology.orchestra.run.smithi078.stdout: "http://smithi051.front.sepia.ceph.com:8000" 2020-11-09T18:18:18.941 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:18:18.941 INFO:teuthology.orchestra.run.smithi078.stdout: "hostnames": [], 2020-11-09T18:18:18.942 INFO:teuthology.orchestra.run.smithi078.stdout: "hostnames_s3website": [], 2020-11-09T18:18:18.942 INFO:teuthology.orchestra.run.smithi078.stdout: "master_zone": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:18:18.942 INFO:teuthology.orchestra.run.smithi078.stdout: "zones": [ 2020-11-09T18:18:18.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:18.942 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:18:18.942 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "test-zone1", 2020-11-09T18:18:18.942 INFO:teuthology.orchestra.run.smithi078.stdout: "endpoints": [ 2020-11-09T18:18:18.943 INFO:teuthology.orchestra.run.smithi078.stdout: "http://smithi051.front.sepia.ceph.com:8000" 2020-11-09T18:18:18.943 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:18:18.943 INFO:teuthology.orchestra.run.smithi078.stdout: "log_meta": "false", 2020-11-09T18:18:18.943 INFO:teuthology.orchestra.run.smithi078.stdout: "log_data": "true", 2020-11-09T18:18:18.943 INFO:teuthology.orchestra.run.smithi078.stdout: "bucket_index_max_shards": 11, 2020-11-09T18:18:18.943 INFO:teuthology.orchestra.run.smithi078.stdout: "read_only": "false", 2020-11-09T18:18:18.943 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_type": "", 2020-11-09T18:18:18.944 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_from_all": "true", 2020-11-09T18:18:18.944 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_from": [], 2020-11-09T18:18:18.944 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect_zone": "" 2020-11-09T18:18:18.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:18.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:18.944 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "1fe6c3d0-3a82-4fac-b1e9-7bfe3c75b156", 2020-11-09T18:18:18.944 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "remove", 2020-11-09T18:18:18.945 INFO:teuthology.orchestra.run.smithi078.stdout: "endpoints": [], 2020-11-09T18:18:18.945 INFO:teuthology.orchestra.run.smithi078.stdout: "log_meta": "false", 2020-11-09T18:18:18.945 INFO:teuthology.orchestra.run.smithi078.stdout: "log_data": "true", 2020-11-09T18:18:18.945 INFO:teuthology.orchestra.run.smithi078.stdout: "bucket_index_max_shards": 11, 2020-11-09T18:18:18.945 INFO:teuthology.orchestra.run.smithi078.stdout: "read_only": "false", 2020-11-09T18:18:18.945 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_type": "", 2020-11-09T18:18:18.945 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_from_all": "true", 2020-11-09T18:18:18.946 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_from": [], 2020-11-09T18:18:18.946 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect_zone": "" 2020-11-09T18:18:18.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:18.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:18.946 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "4600c141-233a-40b8-a0db-e3e3a34df6c3", 2020-11-09T18:18:18.946 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "test-zone4", 2020-11-09T18:18:18.946 INFO:teuthology.orchestra.run.smithi078.stdout: "endpoints": [ 2020-11-09T18:18:18.947 INFO:teuthology.orchestra.run.smithi078.stdout: "http://smithi078.front.sepia.ceph.com:8001" 2020-11-09T18:18:18.947 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:18:18.947 INFO:teuthology.orchestra.run.smithi078.stdout: "log_meta": "false", 2020-11-09T18:18:18.947 INFO:teuthology.orchestra.run.smithi078.stdout: "log_data": "true", 2020-11-09T18:18:18.947 INFO:teuthology.orchestra.run.smithi078.stdout: "bucket_index_max_shards": 11, 2020-11-09T18:18:18.947 INFO:teuthology.orchestra.run.smithi078.stdout: "read_only": "false", 2020-11-09T18:18:18.947 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_type": "pubsub", 2020-11-09T18:18:18.947 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_from_all": "false", 2020-11-09T18:18:18.948 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_from": [ 2020-11-09T18:18:18.948 INFO:teuthology.orchestra.run.smithi078.stdout: "test-zone1" 2020-11-09T18:18:18.948 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:18:18.948 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect_zone": "" 2020-11-09T18:18:18.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:18.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:18.948 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc", 2020-11-09T18:18:18.949 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "test-zone2", 2020-11-09T18:18:18.949 INFO:teuthology.orchestra.run.smithi078.stdout: "endpoints": [ 2020-11-09T18:18:18.949 INFO:teuthology.orchestra.run.smithi078.stdout: "http://smithi078.front.sepia.ceph.com:8000" 2020-11-09T18:18:18.949 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:18:18.949 INFO:teuthology.orchestra.run.smithi078.stdout: "log_meta": "false", 2020-11-09T18:18:18.949 INFO:teuthology.orchestra.run.smithi078.stdout: "log_data": "true", 2020-11-09T18:18:18.949 INFO:teuthology.orchestra.run.smithi078.stdout: "bucket_index_max_shards": 11, 2020-11-09T18:18:18.950 INFO:teuthology.orchestra.run.smithi078.stdout: "read_only": "false", 2020-11-09T18:18:18.950 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_type": "", 2020-11-09T18:18:18.950 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_from_all": "false", 2020-11-09T18:18:18.950 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_from": [], 2020-11-09T18:18:18.950 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect_zone": "" 2020-11-09T18:18:18.950 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:18.950 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:18.951 INFO:teuthology.orchestra.run.smithi078.stdout: "id": "a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6", 2020-11-09T18:18:18.951 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "test-zone3", 2020-11-09T18:18:18.951 INFO:teuthology.orchestra.run.smithi078.stdout: "endpoints": [ 2020-11-09T18:18:18.951 INFO:teuthology.orchestra.run.smithi078.stdout: "http://smithi051.front.sepia.ceph.com:8001" 2020-11-09T18:18:18.951 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:18:18.951 INFO:teuthology.orchestra.run.smithi078.stdout: "log_meta": "false", 2020-11-09T18:18:18.951 INFO:teuthology.orchestra.run.smithi078.stdout: "log_data": "true", 2020-11-09T18:18:18.951 INFO:teuthology.orchestra.run.smithi078.stdout: "bucket_index_max_shards": 11, 2020-11-09T18:18:18.952 INFO:teuthology.orchestra.run.smithi078.stdout: "read_only": "false", 2020-11-09T18:18:18.952 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_type": "", 2020-11-09T18:18:18.952 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_from_all": "true", 2020-11-09T18:18:18.952 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_from": [], 2020-11-09T18:18:18.952 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect_zone": "" 2020-11-09T18:18:18.952 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:18.962 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:18:18.962 INFO:teuthology.orchestra.run.smithi078.stdout: "placement_targets": [ 2020-11-09T18:18:18.963 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:18.963 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "default-placement", 2020-11-09T18:18:18.963 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2020-11-09T18:18:18.963 INFO:teuthology.orchestra.run.smithi078.stdout: "storage_classes": [ 2020-11-09T18:18:18.963 INFO:teuthology.orchestra.run.smithi078.stdout: "STANDARD" 2020-11-09T18:18:18.963 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:18:18.964 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:18.964 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:18:18.964 INFO:teuthology.orchestra.run.smithi078.stdout: "default_placement": "default-placement", 2020-11-09T18:18:18.964 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_id": "0de9c5fb-e243-4abd-b414-59ffbc2cb575", 2020-11-09T18:18:18.965 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_policy": { 2020-11-09T18:18:18.965 INFO:teuthology.orchestra.run.smithi078.stdout: "groups": [] 2020-11-09T18:18:18.965 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:18.965 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:18.965 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2020-11-09T18:18:18.965 INFO:teuthology.orchestra.run.smithi078.stdout: "short_zone_ids": [ 2020-11-09T18:18:18.965 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:18.965 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:18:18.966 INFO:teuthology.orchestra.run.smithi078.stdout: "val": 2019612975 2020-11-09T18:18:18.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:18.966 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:18.966 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "1fe6c3d0-3a82-4fac-b1e9-7bfe3c75b156", 2020-11-09T18:18:18.966 INFO:teuthology.orchestra.run.smithi078.stdout: "val": 2440664360 2020-11-09T18:18:18.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:18.967 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:18.967 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "4600c141-233a-40b8-a0db-e3e3a34df6c3", 2020-11-09T18:18:18.967 INFO:teuthology.orchestra.run.smithi078.stdout: "val": 1720005997 2020-11-09T18:18:18.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:18.967 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:18.967 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc", 2020-11-09T18:18:18.967 INFO:teuthology.orchestra.run.smithi078.stdout: "val": 2281443694 2020-11-09T18:18:18.968 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:18.968 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:18.968 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6", 2020-11-09T18:18:18.968 INFO:teuthology.orchestra.run.smithi078.stdout: "val": 648291153 2020-11-09T18:18:18.968 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:18.968 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:18:18.968 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:18.969 INFO:teuthology.orchestra.run.smithi078.stdout: "master_zonegroup": "test-zonegroup", 2020-11-09T18:18:18.969 INFO:teuthology.orchestra.run.smithi078.stdout: "master_zone": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:18:18.969 INFO:teuthology.orchestra.run.smithi078.stdout: "period_config": { 2020-11-09T18:18:18.969 INFO:teuthology.orchestra.run.smithi078.stdout: "bucket_quota": { 2020-11-09T18:18:18.970 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": false, 2020-11-09T18:18:18.970 INFO:teuthology.orchestra.run.smithi078.stdout: "check_on_raw": false, 2020-11-09T18:18:18.970 INFO:teuthology.orchestra.run.smithi078.stdout: "max_size": -1, 2020-11-09T18:18:18.972 INFO:teuthology.orchestra.run.smithi078.stdout: "max_size_kb": 0, 2020-11-09T18:18:18.972 INFO:teuthology.orchestra.run.smithi078.stdout: "max_objects": -1 2020-11-09T18:18:18.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:18.973 INFO:teuthology.orchestra.run.smithi078.stdout: "user_quota": { 2020-11-09T18:18:18.973 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": false, 2020-11-09T18:18:18.973 INFO:teuthology.orchestra.run.smithi078.stdout: "check_on_raw": false, 2020-11-09T18:18:18.973 INFO:teuthology.orchestra.run.smithi078.stdout: "max_size": -1, 2020-11-09T18:18:18.973 INFO:teuthology.orchestra.run.smithi078.stdout: "max_size_kb": 0, 2020-11-09T18:18:18.973 INFO:teuthology.orchestra.run.smithi078.stdout: "max_objects": -1 2020-11-09T18:18:18.973 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:18.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:18.974 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_id": "0de9c5fb-e243-4abd-b414-59ffbc2cb575", 2020-11-09T18:18:18.974 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_name": "test-realm", 2020-11-09T18:18:18.974 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:18:18.974 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:18:18.975 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:18:18.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:18:18.976 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-09T18:18:19.017 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:18:19.017 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:18:19.039 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:18:19.033+0000 7fdc1132e0c0 0 failed reading obj info from .rgw.root:zone_info.4600c141-233a-40b8-a0db-e3e3a34df6c3: (2) No such file or directory 2020-11-09T18:18:19.039 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:18:19.033+0000 7fdc1132e0c0 0 WARNING: could not read zone params for zone id=4600c141-233a-40b8-a0db-e3e3a34df6c3 name=test-zone4 2020-11-09T18:18:19.040 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:18:19.033+0000 7fdc1132e0c0 0 failed reading obj info from .rgw.root:zone_info.4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc: (2) No such file or directory 2020-11-09T18:18:19.040 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:18:19.033+0000 7fdc1132e0c0 0 WARNING: could not read zone params for zone id=4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc name=test-zone2 2020-11-09T18:18:19.047 INFO:teuthology.orchestra.run.smithi051.stdout:{ 2020-11-09T18:18:19.047 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "test-zonegroup", 2020-11-09T18:18:19.048 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "test-zonegroup", 2020-11-09T18:18:19.048 INFO:teuthology.orchestra.run.smithi051.stdout: "api_name": "", 2020-11-09T18:18:19.048 INFO:teuthology.orchestra.run.smithi051.stdout: "is_master": "true", 2020-11-09T18:18:19.048 INFO:teuthology.orchestra.run.smithi051.stdout: "endpoints": [ 2020-11-09T18:18:19.048 INFO:teuthology.orchestra.run.smithi051.stdout: "http://smithi051.front.sepia.ceph.com:8000" 2020-11-09T18:18:19.048 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:18:19.049 INFO:teuthology.orchestra.run.smithi051.stdout: "hostnames": [], 2020-11-09T18:18:19.049 INFO:teuthology.orchestra.run.smithi051.stdout: "hostnames_s3website": [], 2020-11-09T18:18:19.049 INFO:teuthology.orchestra.run.smithi051.stdout: "master_zone": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:18:19.049 INFO:teuthology.orchestra.run.smithi051.stdout: "zones": [ 2020-11-09T18:18:19.049 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:19.049 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:18:19.049 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "test-zone1", 2020-11-09T18:18:19.050 INFO:teuthology.orchestra.run.smithi051.stdout: "endpoints": [ 2020-11-09T18:18:19.050 INFO:teuthology.orchestra.run.smithi051.stdout: "http://smithi051.front.sepia.ceph.com:8000" 2020-11-09T18:18:19.050 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:18:19.050 INFO:teuthology.orchestra.run.smithi051.stdout: "log_meta": "false", 2020-11-09T18:18:19.050 INFO:teuthology.orchestra.run.smithi051.stdout: "log_data": "true", 2020-11-09T18:18:19.050 INFO:teuthology.orchestra.run.smithi051.stdout: "bucket_index_max_shards": 11, 2020-11-09T18:18:19.050 INFO:teuthology.orchestra.run.smithi051.stdout: "read_only": "false", 2020-11-09T18:18:19.051 INFO:teuthology.orchestra.run.smithi051.stdout: "tier_type": "", 2020-11-09T18:18:19.051 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from_all": "true", 2020-11-09T18:18:19.051 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from": [], 2020-11-09T18:18:19.051 INFO:teuthology.orchestra.run.smithi051.stdout: "redirect_zone": "" 2020-11-09T18:18:19.051 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:18:19.051 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:19.051 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "4600c141-233a-40b8-a0db-e3e3a34df6c3", 2020-11-09T18:18:19.052 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "test-zone4", 2020-11-09T18:18:19.052 INFO:teuthology.orchestra.run.smithi051.stdout: "endpoints": [ 2020-11-09T18:18:19.052 INFO:teuthology.orchestra.run.smithi051.stdout: "http://smithi078.front.sepia.ceph.com:8001" 2020-11-09T18:18:19.052 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:18:19.052 INFO:teuthology.orchestra.run.smithi051.stdout: "log_meta": "false", 2020-11-09T18:18:19.052 INFO:teuthology.orchestra.run.smithi051.stdout: "log_data": "true", 2020-11-09T18:18:19.052 INFO:teuthology.orchestra.run.smithi051.stdout: "bucket_index_max_shards": 11, 2020-11-09T18:18:19.053 INFO:teuthology.orchestra.run.smithi051.stdout: "read_only": "false", 2020-11-09T18:18:19.053 INFO:teuthology.orchestra.run.smithi051.stdout: "tier_type": "pubsub", 2020-11-09T18:18:19.053 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from_all": "false", 2020-11-09T18:18:19.053 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from": [ 2020-11-09T18:18:19.053 INFO:teuthology.orchestra.run.smithi051.stdout: "test-zone1" 2020-11-09T18:18:19.053 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:18:19.053 INFO:teuthology.orchestra.run.smithi051.stdout: "redirect_zone": "" 2020-11-09T18:18:19.053 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:18:19.054 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:19.054 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc", 2020-11-09T18:18:19.054 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "test-zone2", 2020-11-09T18:18:19.054 INFO:teuthology.orchestra.run.smithi051.stdout: "endpoints": [ 2020-11-09T18:18:19.054 INFO:teuthology.orchestra.run.smithi051.stdout: "http://smithi078.front.sepia.ceph.com:8000" 2020-11-09T18:18:19.054 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:18:19.055 INFO:teuthology.orchestra.run.smithi051.stdout: "log_meta": "false", 2020-11-09T18:18:19.055 INFO:teuthology.orchestra.run.smithi051.stdout: "log_data": "true", 2020-11-09T18:18:19.055 INFO:teuthology.orchestra.run.smithi051.stdout: "bucket_index_max_shards": 11, 2020-11-09T18:18:19.055 INFO:teuthology.orchestra.run.smithi051.stdout: "read_only": "false", 2020-11-09T18:18:19.055 INFO:teuthology.orchestra.run.smithi051.stdout: "tier_type": "", 2020-11-09T18:18:19.055 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from_all": "false", 2020-11-09T18:18:19.055 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from": [], 2020-11-09T18:18:19.056 INFO:teuthology.orchestra.run.smithi051.stdout: "redirect_zone": "" 2020-11-09T18:18:19.056 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:18:19.056 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:19.056 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6", 2020-11-09T18:18:19.056 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "test-zone3", 2020-11-09T18:18:19.056 INFO:teuthology.orchestra.run.smithi051.stdout: "endpoints": [ 2020-11-09T18:18:19.056 INFO:teuthology.orchestra.run.smithi051.stdout: "http://smithi051.front.sepia.ceph.com:8001" 2020-11-09T18:18:19.057 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:18:19.057 INFO:teuthology.orchestra.run.smithi051.stdout: "log_meta": "false", 2020-11-09T18:18:19.057 INFO:teuthology.orchestra.run.smithi051.stdout: "log_data": "true", 2020-11-09T18:18:19.057 INFO:teuthology.orchestra.run.smithi051.stdout: "bucket_index_max_shards": 11, 2020-11-09T18:18:19.058 INFO:teuthology.orchestra.run.smithi051.stdout: "read_only": "false", 2020-11-09T18:18:19.058 INFO:teuthology.orchestra.run.smithi051.stdout: "tier_type": "", 2020-11-09T18:18:19.058 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from_all": "true", 2020-11-09T18:18:19.058 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from": [], 2020-11-09T18:18:19.058 INFO:teuthology.orchestra.run.smithi051.stdout: "redirect_zone": "" 2020-11-09T18:18:19.058 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:18:19.059 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:18:19.059 INFO:teuthology.orchestra.run.smithi051.stdout: "placement_targets": [ 2020-11-09T18:18:19.059 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:19.059 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "default-placement", 2020-11-09T18:18:19.059 INFO:teuthology.orchestra.run.smithi051.stdout: "tags": [], 2020-11-09T18:18:19.059 INFO:teuthology.orchestra.run.smithi051.stdout: "storage_classes": [ 2020-11-09T18:18:19.059 INFO:teuthology.orchestra.run.smithi051.stdout: "STANDARD" 2020-11-09T18:18:19.060 INFO:teuthology.orchestra.run.smithi051.stdout: ] 2020-11-09T18:18:19.060 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:18:19.060 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:18:19.060 INFO:teuthology.orchestra.run.smithi051.stdout: "default_placement": "default-placement", 2020-11-09T18:18:19.060 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_id": "0de9c5fb-e243-4abd-b414-59ffbc2cb575", 2020-11-09T18:18:19.060 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_policy": { 2020-11-09T18:18:19.060 INFO:teuthology.orchestra.run.smithi051.stdout: "groups": [] 2020-11-09T18:18:19.061 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:18:19.061 INFO:teuthology.orchestra.run.smithi051.stdout:} 2020-11-09T18:18:19.062 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:18:19.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:18:19.063 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-09T18:18:19.145 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:18:19.146 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:18:19.165 INFO:teuthology.orchestra.run.smithi051.stderr:zone name remove not found in zonegroup test-zonegroup 2020-11-09T18:18:19.174 DEBUG:teuthology.orchestra.run:got remote process result: 2 2020-11-09T18:18:19.175 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:18:19.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:18:19.176 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone delete --rgw-zone remove --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-09T18:18:19.214 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:18:19.215 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:18:19.256 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:18:19.249+0000 7f0364b8a0c0 0 failed reading obj info from .rgw.root:zone_info.11f516a7-3288-49a6-9a89-b45ac1dae91a: (2) No such file or directory 2020-11-09T18:18:19.256 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:18:19.249+0000 7f0364b8a0c0 0 WARNING: could not read zone params for zone id=11f516a7-3288-49a6-9a89-b45ac1dae91a name=test-zone1 2020-11-09T18:18:19.257 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:18:19.253+0000 7f0364b8a0c0 0 failed reading obj info from .rgw.root:zone_info.a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6: (2) No such file or directory 2020-11-09T18:18:19.258 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:18:19.253+0000 7f0364b8a0c0 0 WARNING: could not read zone params for zone id=a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6 name=test-zone3 2020-11-09T18:18:19.262 INFO:teuthology.orchestra.run.smithi078.stderr:2020-11-09T18:18:19.257+0000 7f0364b8a0c0 0 zone id 1fe6c3d0-3a82-4fac-b1e9-7bfe3c75b156 is not a part of zonegroup default 2020-11-09T18:18:19.273 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:18:19.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:18:19.275 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-09T18:18:19.323 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:18:19.324 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:18:19.397 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:18:19.393+0000 7f6d6dbec0c0 0 WARNING: can't generate connection for zone 1fe6c3d0-3a82-4fac-b1e9-7bfe3c75b156 id remove: no endpoints defined 2020-11-09T18:18:19.769 INFO:teuthology.orchestra.run.smithi051.stdout:{ 2020-11-09T18:18:19.770 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:18:19.770 INFO:teuthology.orchestra.run.smithi051.stdout: "epoch": 4, 2020-11-09T18:18:19.770 INFO:teuthology.orchestra.run.smithi051.stdout: "predecessor_uuid": "bbf9f070-21b9-4940-95e9-d031ba1c647f", 2020-11-09T18:18:19.770 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_status": [ 2020-11-09T18:18:19.771 INFO:teuthology.orchestra.run.smithi051.stdout: "1_1604945680.637050_159.1", 2020-11-09T18:18:19.771 INFO:teuthology.orchestra.run.smithi051.stdout: "", 2020-11-09T18:18:19.771 INFO:teuthology.orchestra.run.smithi051.stdout: "", 2020-11-09T18:18:19.771 INFO:teuthology.orchestra.run.smithi051.stdout: "" 2020-11-09T18:18:19.771 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:18:19.771 INFO:teuthology.orchestra.run.smithi051.stdout: "period_map": { 2020-11-09T18:18:19.771 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:18:19.771 INFO:teuthology.orchestra.run.smithi051.stdout: "zonegroups": [ 2020-11-09T18:18:19.772 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:19.772 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "test-zonegroup", 2020-11-09T18:18:19.772 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "test-zonegroup", 2020-11-09T18:18:19.772 INFO:teuthology.orchestra.run.smithi051.stdout: "api_name": "", 2020-11-09T18:18:19.772 INFO:teuthology.orchestra.run.smithi051.stdout: "is_master": "true", 2020-11-09T18:18:19.772 INFO:teuthology.orchestra.run.smithi051.stdout: "endpoints": [ 2020-11-09T18:18:19.773 INFO:teuthology.orchestra.run.smithi051.stdout: "http://smithi051.front.sepia.ceph.com:8000" 2020-11-09T18:18:19.773 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:18:19.773 INFO:teuthology.orchestra.run.smithi051.stdout: "hostnames": [], 2020-11-09T18:18:19.773 INFO:teuthology.orchestra.run.smithi051.stdout: "hostnames_s3website": [], 2020-11-09T18:18:19.773 INFO:teuthology.orchestra.run.smithi051.stdout: "master_zone": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:18:19.773 INFO:teuthology.orchestra.run.smithi051.stdout: "zones": [ 2020-11-09T18:18:19.773 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:19.774 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:18:19.774 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "test-zone1", 2020-11-09T18:18:19.774 INFO:teuthology.orchestra.run.smithi051.stdout: "endpoints": [ 2020-11-09T18:18:19.774 INFO:teuthology.orchestra.run.smithi051.stdout: "http://smithi051.front.sepia.ceph.com:8000" 2020-11-09T18:18:19.774 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:18:19.774 INFO:teuthology.orchestra.run.smithi051.stdout: "log_meta": "false", 2020-11-09T18:18:19.774 INFO:teuthology.orchestra.run.smithi051.stdout: "log_data": "true", 2020-11-09T18:18:19.774 INFO:teuthology.orchestra.run.smithi051.stdout: "bucket_index_max_shards": 11, 2020-11-09T18:18:19.775 INFO:teuthology.orchestra.run.smithi051.stdout: "read_only": "false", 2020-11-09T18:18:19.775 INFO:teuthology.orchestra.run.smithi051.stdout: "tier_type": "", 2020-11-09T18:18:19.775 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from_all": "true", 2020-11-09T18:18:19.775 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from": [], 2020-11-09T18:18:19.775 INFO:teuthology.orchestra.run.smithi051.stdout: "redirect_zone": "" 2020-11-09T18:18:19.775 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:18:19.775 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:19.776 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "4600c141-233a-40b8-a0db-e3e3a34df6c3", 2020-11-09T18:18:19.776 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "test-zone4", 2020-11-09T18:18:19.776 INFO:teuthology.orchestra.run.smithi051.stdout: "endpoints": [ 2020-11-09T18:18:19.776 INFO:teuthology.orchestra.run.smithi051.stdout: "http://smithi078.front.sepia.ceph.com:8001" 2020-11-09T18:18:19.776 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:18:19.776 INFO:teuthology.orchestra.run.smithi051.stdout: "log_meta": "false", 2020-11-09T18:18:19.776 INFO:teuthology.orchestra.run.smithi051.stdout: "log_data": "true", 2020-11-09T18:18:19.777 INFO:teuthology.orchestra.run.smithi051.stdout: "bucket_index_max_shards": 11, 2020-11-09T18:18:19.777 INFO:teuthology.orchestra.run.smithi051.stdout: "read_only": "false", 2020-11-09T18:18:19.777 INFO:teuthology.orchestra.run.smithi051.stdout: "tier_type": "pubsub", 2020-11-09T18:18:19.777 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from_all": "false", 2020-11-09T18:18:19.777 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from": [ 2020-11-09T18:18:19.777 INFO:teuthology.orchestra.run.smithi051.stdout: "test-zone1" 2020-11-09T18:18:19.777 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:18:19.778 INFO:teuthology.orchestra.run.smithi051.stdout: "redirect_zone": "" 2020-11-09T18:18:19.778 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:18:19.778 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:19.778 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc", 2020-11-09T18:18:19.778 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "test-zone2", 2020-11-09T18:18:19.778 INFO:teuthology.orchestra.run.smithi051.stdout: "endpoints": [ 2020-11-09T18:18:19.778 INFO:teuthology.orchestra.run.smithi051.stdout: "http://smithi078.front.sepia.ceph.com:8000" 2020-11-09T18:18:19.778 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:18:19.779 INFO:teuthology.orchestra.run.smithi051.stdout: "log_meta": "false", 2020-11-09T18:18:19.779 INFO:teuthology.orchestra.run.smithi051.stdout: "log_data": "true", 2020-11-09T18:18:19.779 INFO:teuthology.orchestra.run.smithi051.stdout: "bucket_index_max_shards": 11, 2020-11-09T18:18:19.779 INFO:teuthology.orchestra.run.smithi051.stdout: "read_only": "false", 2020-11-09T18:18:19.779 INFO:teuthology.orchestra.run.smithi051.stdout: "tier_type": "", 2020-11-09T18:18:19.779 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from_all": "false", 2020-11-09T18:18:19.779 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from": [], 2020-11-09T18:18:19.780 INFO:teuthology.orchestra.run.smithi051.stdout: "redirect_zone": "" 2020-11-09T18:18:19.780 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:18:19.780 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:19.780 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6", 2020-11-09T18:18:19.780 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "test-zone3", 2020-11-09T18:18:19.780 INFO:teuthology.orchestra.run.smithi051.stdout: "endpoints": [ 2020-11-09T18:18:19.780 INFO:teuthology.orchestra.run.smithi051.stdout: "http://smithi051.front.sepia.ceph.com:8001" 2020-11-09T18:18:19.781 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:18:19.781 INFO:teuthology.orchestra.run.smithi051.stdout: "log_meta": "false", 2020-11-09T18:18:19.781 INFO:teuthology.orchestra.run.smithi051.stdout: "log_data": "true", 2020-11-09T18:18:19.781 INFO:teuthology.orchestra.run.smithi051.stdout: "bucket_index_max_shards": 11, 2020-11-09T18:18:19.781 INFO:teuthology.orchestra.run.smithi051.stdout: "read_only": "false", 2020-11-09T18:18:19.781 INFO:teuthology.orchestra.run.smithi051.stdout: "tier_type": "", 2020-11-09T18:18:19.781 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from_all": "true", 2020-11-09T18:18:19.781 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from": [], 2020-11-09T18:18:19.782 INFO:teuthology.orchestra.run.smithi051.stdout: "redirect_zone": "" 2020-11-09T18:18:19.782 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:18:19.782 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:18:19.782 INFO:teuthology.orchestra.run.smithi051.stdout: "placement_targets": [ 2020-11-09T18:18:19.782 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:19.782 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "default-placement", 2020-11-09T18:18:19.782 INFO:teuthology.orchestra.run.smithi051.stdout: "tags": [], 2020-11-09T18:18:19.783 INFO:teuthology.orchestra.run.smithi051.stdout: "storage_classes": [ 2020-11-09T18:18:19.783 INFO:teuthology.orchestra.run.smithi051.stdout: "STANDARD" 2020-11-09T18:18:19.783 INFO:teuthology.orchestra.run.smithi051.stdout: ] 2020-11-09T18:18:19.783 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:18:19.783 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:18:19.783 INFO:teuthology.orchestra.run.smithi051.stdout: "default_placement": "default-placement", 2020-11-09T18:18:19.783 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_id": "0de9c5fb-e243-4abd-b414-59ffbc2cb575", 2020-11-09T18:18:19.783 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_policy": { 2020-11-09T18:18:19.784 INFO:teuthology.orchestra.run.smithi051.stdout: "groups": [] 2020-11-09T18:18:19.784 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:18:19.784 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:18:19.784 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:18:19.828 INFO:teuthology.orchestra.run.smithi051.stdout: "short_zone_ids": [ 2020-11-09T18:18:19.829 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:19.829 INFO:teuthology.orchestra.run.smithi051.stdout: "key": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:18:19.829 INFO:teuthology.orchestra.run.smithi051.stdout: "val": 2019612975 2020-11-09T18:18:19.829 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:18:19.830 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:19.830 INFO:teuthology.orchestra.run.smithi051.stdout: "key": "4600c141-233a-40b8-a0db-e3e3a34df6c3", 2020-11-09T18:18:19.830 INFO:teuthology.orchestra.run.smithi051.stdout: "val": 1720005997 2020-11-09T18:18:19.830 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:18:19.830 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:19.831 INFO:teuthology.orchestra.run.smithi051.stdout: "key": "4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc", 2020-11-09T18:18:19.831 INFO:teuthology.orchestra.run.smithi051.stdout: "val": 2281443694 2020-11-09T18:18:19.831 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:18:19.831 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:19.831 INFO:teuthology.orchestra.run.smithi051.stdout: "key": "a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6", 2020-11-09T18:18:19.831 INFO:teuthology.orchestra.run.smithi051.stdout: "val": 648291153 2020-11-09T18:18:19.832 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:18:19.832 INFO:teuthology.orchestra.run.smithi051.stdout: ] 2020-11-09T18:18:19.832 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:18:19.832 INFO:teuthology.orchestra.run.smithi051.stdout: "master_zonegroup": "test-zonegroup", 2020-11-09T18:18:19.832 INFO:teuthology.orchestra.run.smithi051.stdout: "master_zone": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:18:19.833 INFO:teuthology.orchestra.run.smithi051.stdout: "period_config": { 2020-11-09T18:18:19.833 INFO:teuthology.orchestra.run.smithi051.stdout: "bucket_quota": { 2020-11-09T18:18:19.833 INFO:teuthology.orchestra.run.smithi051.stdout: "enabled": false, 2020-11-09T18:18:19.833 INFO:teuthology.orchestra.run.smithi051.stdout: "check_on_raw": false, 2020-11-09T18:18:19.833 INFO:teuthology.orchestra.run.smithi051.stdout: "max_size": -1, 2020-11-09T18:18:19.833 INFO:teuthology.orchestra.run.smithi051.stdout: "max_size_kb": 0, 2020-11-09T18:18:19.834 INFO:teuthology.orchestra.run.smithi051.stdout: "max_objects": -1 2020-11-09T18:18:19.834 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:18:19.834 INFO:teuthology.orchestra.run.smithi051.stdout: "user_quota": { 2020-11-09T18:18:19.834 INFO:teuthology.orchestra.run.smithi051.stdout: "enabled": false, 2020-11-09T18:18:19.834 INFO:teuthology.orchestra.run.smithi051.stdout: "check_on_raw": false, 2020-11-09T18:18:19.835 INFO:teuthology.orchestra.run.smithi051.stdout: "max_size": -1, 2020-11-09T18:18:19.835 INFO:teuthology.orchestra.run.smithi051.stdout: "max_size_kb": 0, 2020-11-09T18:18:19.835 INFO:teuthology.orchestra.run.smithi051.stdout: "max_objects": -1 2020-11-09T18:18:19.835 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:18:19.835 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:18:19.835 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_id": "0de9c5fb-e243-4abd-b414-59ffbc2cb575", 2020-11-09T18:18:19.836 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_name": "test-realm", 2020-11-09T18:18:19.836 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 4 2020-11-09T18:18:19.836 INFO:teuthology.orchestra.run.smithi051.stdout:} 2020-11-09T18:18:19.837 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_zonegroup_remove ... ok 2020-11-09T18:18:19.838 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:18:19.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:18:19.839 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-09T18:18:19.872 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:18:19.873 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:18:19.904 INFO:teuthology.orchestra.run.smithi051.stdout:{ 2020-11-09T18:18:19.904 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "001e22e0-6a17-43b0-b568-f8e97686e850", 2020-11-09T18:18:19.905 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "remove_zg", 2020-11-09T18:18:19.905 INFO:teuthology.orchestra.run.smithi051.stdout: "api_name": "remove_zg", 2020-11-09T18:18:19.905 INFO:teuthology.orchestra.run.smithi051.stdout: "is_master": "false", 2020-11-09T18:18:19.906 INFO:teuthology.orchestra.run.smithi051.stdout: "endpoints": [], 2020-11-09T18:18:19.906 INFO:teuthology.orchestra.run.smithi051.stdout: "hostnames": [], 2020-11-09T18:18:19.906 INFO:teuthology.orchestra.run.smithi051.stdout: "hostnames_s3website": [], 2020-11-09T18:18:19.907 INFO:teuthology.orchestra.run.smithi051.stdout: "master_zone": "", 2020-11-09T18:18:19.907 INFO:teuthology.orchestra.run.smithi051.stdout: "zones": [], 2020-11-09T18:18:19.907 INFO:teuthology.orchestra.run.smithi051.stdout: "placement_targets": [], 2020-11-09T18:18:19.907 INFO:teuthology.orchestra.run.smithi051.stdout: "default_placement": "", 2020-11-09T18:18:19.908 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_id": "0de9c5fb-e243-4abd-b414-59ffbc2cb575", 2020-11-09T18:18:19.908 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_policy": { 2020-11-09T18:18:19.908 INFO:teuthology.orchestra.run.smithi051.stdout: "groups": [] 2020-11-09T18:18:19.909 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:18:19.909 INFO:teuthology.orchestra.run.smithi051.stdout:} 2020-11-09T18:18:19.910 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:18:19.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:18:19.912 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-09T18:18:19.991 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:18:19.991 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:18:20.065 INFO:teuthology.orchestra.run.smithi051.stdout:{ 2020-11-09T18:18:20.066 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "4d0e43a6-0ddb-4d76-979c-891a9abfd6b4", 2020-11-09T18:18:20.066 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "remove", 2020-11-09T18:18:20.066 INFO:teuthology.orchestra.run.smithi051.stdout: "domain_root": "remove.rgw.meta:root", 2020-11-09T18:18:20.067 INFO:teuthology.orchestra.run.smithi051.stdout: "control_pool": "remove.rgw.control", 2020-11-09T18:18:20.067 INFO:teuthology.orchestra.run.smithi051.stdout: "gc_pool": "remove.rgw.log:gc", 2020-11-09T18:18:20.067 INFO:teuthology.orchestra.run.smithi051.stdout: "lc_pool": "remove.rgw.log:lc", 2020-11-09T18:18:20.068 INFO:teuthology.orchestra.run.smithi051.stdout: "log_pool": "remove.rgw.log", 2020-11-09T18:18:20.068 INFO:teuthology.orchestra.run.smithi051.stdout: "intent_log_pool": "remove.rgw.log:intent", 2020-11-09T18:18:20.068 INFO:teuthology.orchestra.run.smithi051.stdout: "usage_log_pool": "remove.rgw.log:usage", 2020-11-09T18:18:20.068 INFO:teuthology.orchestra.run.smithi051.stdout: "roles_pool": "remove.rgw.meta:roles", 2020-11-09T18:18:20.069 INFO:teuthology.orchestra.run.smithi051.stdout: "reshard_pool": "remove.rgw.log:reshard", 2020-11-09T18:18:20.069 INFO:teuthology.orchestra.run.smithi051.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2020-11-09T18:18:20.069 INFO:teuthology.orchestra.run.smithi051.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2020-11-09T18:18:20.070 INFO:teuthology.orchestra.run.smithi051.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2020-11-09T18:18:20.070 INFO:teuthology.orchestra.run.smithi051.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2020-11-09T18:18:20.071 INFO:teuthology.orchestra.run.smithi051.stdout: "otp_pool": "remove.rgw.otp", 2020-11-09T18:18:20.071 INFO:teuthology.orchestra.run.smithi051.stdout: "system_key": { 2020-11-09T18:18:20.071 INFO:teuthology.orchestra.run.smithi051.stdout: "access_key": "", 2020-11-09T18:18:20.071 INFO:teuthology.orchestra.run.smithi051.stdout: "secret_key": "" 2020-11-09T18:18:20.071 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:18:20.072 INFO:teuthology.orchestra.run.smithi051.stdout: "placement_pools": [ 2020-11-09T18:18:20.072 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:20.072 INFO:teuthology.orchestra.run.smithi051.stdout: "key": "default-placement", 2020-11-09T18:18:20.072 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:18:20.073 INFO:teuthology.orchestra.run.smithi051.stdout: "index_pool": "remove.rgw.buckets.index", 2020-11-09T18:18:20.073 INFO:teuthology.orchestra.run.smithi051.stdout: "storage_classes": { 2020-11-09T18:18:20.073 INFO:teuthology.orchestra.run.smithi051.stdout: "STANDARD": { 2020-11-09T18:18:20.073 INFO:teuthology.orchestra.run.smithi051.stdout: "data_pool": "remove.rgw.buckets.data" 2020-11-09T18:18:20.074 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:18:20.074 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:18:20.074 INFO:teuthology.orchestra.run.smithi051.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2020-11-09T18:18:20.075 INFO:teuthology.orchestra.run.smithi051.stdout: "index_type": 0 2020-11-09T18:18:20.075 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:18:20.075 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:18:20.075 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:18:20.076 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_id": "0de9c5fb-e243-4abd-b414-59ffbc2cb575", 2020-11-09T18:18:20.076 INFO:teuthology.orchestra.run.smithi051.stdout: "notif_pool": "remove.rgw.log:notif" 2020-11-09T18:18:20.076 INFO:teuthology.orchestra.run.smithi051.stdout:} 2020-11-09T18:18:20.078 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:18:20.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:18:20.080 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-09T18:18:20.166 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:18:20.167 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:18:20.476 INFO:teuthology.orchestra.run.smithi051.stdout:{ 2020-11-09T18:18:20.485 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:18:20.485 INFO:teuthology.orchestra.run.smithi051.stdout: "epoch": 5, 2020-11-09T18:18:20.485 INFO:teuthology.orchestra.run.smithi051.stdout: "predecessor_uuid": "bbf9f070-21b9-4940-95e9-d031ba1c647f", 2020-11-09T18:18:20.485 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_status": [ 2020-11-09T18:18:20.485 INFO:teuthology.orchestra.run.smithi051.stdout: "1_1604945680.637050_159.1", 2020-11-09T18:18:20.485 INFO:teuthology.orchestra.run.smithi051.stdout: "", 2020-11-09T18:18:20.486 INFO:teuthology.orchestra.run.smithi051.stdout: "", 2020-11-09T18:18:20.486 INFO:teuthology.orchestra.run.smithi051.stdout: "" 2020-11-09T18:18:20.486 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:18:20.486 INFO:teuthology.orchestra.run.smithi051.stdout: "period_map": { 2020-11-09T18:18:20.486 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:18:20.486 INFO:teuthology.orchestra.run.smithi051.stdout: "zonegroups": [ 2020-11-09T18:18:20.486 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:20.487 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "001e22e0-6a17-43b0-b568-f8e97686e850", 2020-11-09T18:18:20.487 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "remove_zg", 2020-11-09T18:18:20.491 INFO:teuthology.orchestra.run.smithi051.stdout: "api_name": "remove_zg", 2020-11-09T18:18:20.492 INFO:teuthology.orchestra.run.smithi051.stdout: "is_master": "false", 2020-11-09T18:18:20.492 INFO:teuthology.orchestra.run.smithi051.stdout: "endpoints": [], 2020-11-09T18:18:20.492 INFO:teuthology.orchestra.run.smithi051.stdout: "hostnames": [], 2020-11-09T18:18:20.492 INFO:teuthology.orchestra.run.smithi051.stdout: "hostnames_s3website": [], 2020-11-09T18:18:20.492 INFO:teuthology.orchestra.run.smithi051.stdout: "master_zone": "4d0e43a6-0ddb-4d76-979c-891a9abfd6b4", 2020-11-09T18:18:20.496 INFO:teuthology.orchestra.run.smithi051.stdout: "zones": [ 2020-11-09T18:18:20.497 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:20.497 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "4d0e43a6-0ddb-4d76-979c-891a9abfd6b4", 2020-11-09T18:18:20.497 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "remove", 2020-11-09T18:18:20.497 INFO:teuthology.orchestra.run.smithi051.stdout: "endpoints": [], 2020-11-09T18:18:20.497 INFO:teuthology.orchestra.run.smithi051.stdout: "log_meta": "false", 2020-11-09T18:18:20.497 INFO:teuthology.orchestra.run.smithi051.stdout: "log_data": "false", 2020-11-09T18:18:20.497 INFO:teuthology.orchestra.run.smithi051.stdout: "bucket_index_max_shards": 11, 2020-11-09T18:18:20.498 INFO:teuthology.orchestra.run.smithi051.stdout: "read_only": "false", 2020-11-09T18:18:20.498 INFO:teuthology.orchestra.run.smithi051.stdout: "tier_type": "", 2020-11-09T18:18:20.498 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from_all": "true", 2020-11-09T18:18:20.498 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from": [], 2020-11-09T18:18:20.498 INFO:teuthology.orchestra.run.smithi051.stdout: "redirect_zone": "" 2020-11-09T18:18:20.498 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:18:20.498 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:18:20.499 INFO:teuthology.orchestra.run.smithi051.stdout: "placement_targets": [ 2020-11-09T18:18:20.499 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:20.499 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "default-placement", 2020-11-09T18:18:20.499 INFO:teuthology.orchestra.run.smithi051.stdout: "tags": [], 2020-11-09T18:18:20.499 INFO:teuthology.orchestra.run.smithi051.stdout: "storage_classes": [ 2020-11-09T18:18:20.499 INFO:teuthology.orchestra.run.smithi051.stdout: "STANDARD" 2020-11-09T18:18:20.499 INFO:teuthology.orchestra.run.smithi051.stdout: ] 2020-11-09T18:18:20.500 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:18:20.500 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:18:20.500 INFO:teuthology.orchestra.run.smithi051.stdout: "default_placement": "default-placement", 2020-11-09T18:18:20.500 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_id": "0de9c5fb-e243-4abd-b414-59ffbc2cb575", 2020-11-09T18:18:20.500 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_policy": { 2020-11-09T18:18:20.500 INFO:teuthology.orchestra.run.smithi051.stdout: "groups": [] 2020-11-09T18:18:20.500 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:18:20.501 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:18:20.501 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:20.501 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "test-zonegroup", 2020-11-09T18:18:20.501 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "test-zonegroup", 2020-11-09T18:18:20.501 INFO:teuthology.orchestra.run.smithi051.stdout: "api_name": "", 2020-11-09T18:18:20.501 INFO:teuthology.orchestra.run.smithi051.stdout: "is_master": "true", 2020-11-09T18:18:20.501 INFO:teuthology.orchestra.run.smithi051.stdout: "endpoints": [ 2020-11-09T18:18:20.502 INFO:teuthology.orchestra.run.smithi051.stdout: "http://smithi051.front.sepia.ceph.com:8000" 2020-11-09T18:18:20.502 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:18:20.502 INFO:teuthology.orchestra.run.smithi051.stdout: "hostnames": [], 2020-11-09T18:18:20.502 INFO:teuthology.orchestra.run.smithi051.stdout: "hostnames_s3website": [], 2020-11-09T18:18:20.502 INFO:teuthology.orchestra.run.smithi051.stdout: "master_zone": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:18:20.502 INFO:teuthology.orchestra.run.smithi051.stdout: "zones": [ 2020-11-09T18:18:20.502 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:20.502 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:18:20.503 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "test-zone1", 2020-11-09T18:18:20.503 INFO:teuthology.orchestra.run.smithi051.stdout: "endpoints": [ 2020-11-09T18:18:20.503 INFO:teuthology.orchestra.run.smithi051.stdout: "http://smithi051.front.sepia.ceph.com:8000" 2020-11-09T18:18:20.503 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:18:20.503 INFO:teuthology.orchestra.run.smithi051.stdout: "log_meta": "false", 2020-11-09T18:18:20.504 INFO:teuthology.orchestra.run.smithi051.stdout: "log_data": "true", 2020-11-09T18:18:20.504 INFO:teuthology.orchestra.run.smithi051.stdout: "bucket_index_max_shards": 11, 2020-11-09T18:18:20.504 INFO:teuthology.orchestra.run.smithi051.stdout: "read_only": "false", 2020-11-09T18:18:20.504 INFO:teuthology.orchestra.run.smithi051.stdout: "tier_type": "", 2020-11-09T18:18:20.504 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from_all": "true", 2020-11-09T18:18:20.505 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from": [], 2020-11-09T18:18:20.505 INFO:teuthology.orchestra.run.smithi051.stdout: "redirect_zone": "" 2020-11-09T18:18:20.505 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:18:20.505 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:20.505 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "4600c141-233a-40b8-a0db-e3e3a34df6c3", 2020-11-09T18:18:20.505 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "test-zone4", 2020-11-09T18:18:20.505 INFO:teuthology.orchestra.run.smithi051.stdout: "endpoints": [ 2020-11-09T18:18:20.506 INFO:teuthology.orchestra.run.smithi051.stdout: "http://smithi078.front.sepia.ceph.com:8001" 2020-11-09T18:18:20.506 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:18:20.506 INFO:teuthology.orchestra.run.smithi051.stdout: "log_meta": "false", 2020-11-09T18:18:20.506 INFO:teuthology.orchestra.run.smithi051.stdout: "log_data": "true", 2020-11-09T18:18:20.506 INFO:teuthology.orchestra.run.smithi051.stdout: "bucket_index_max_shards": 11, 2020-11-09T18:18:20.506 INFO:teuthology.orchestra.run.smithi051.stdout: "read_only": "false", 2020-11-09T18:18:20.506 INFO:teuthology.orchestra.run.smithi051.stdout: "tier_type": "pubsub", 2020-11-09T18:18:20.507 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from_all": "false", 2020-11-09T18:18:20.507 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from": [ 2020-11-09T18:18:20.507 INFO:teuthology.orchestra.run.smithi051.stdout: "test-zone1" 2020-11-09T18:18:20.507 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:18:20.507 INFO:teuthology.orchestra.run.smithi051.stdout: "redirect_zone": "" 2020-11-09T18:18:20.507 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:18:20.507 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:20.508 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc", 2020-11-09T18:18:20.508 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "test-zone2", 2020-11-09T18:18:20.508 INFO:teuthology.orchestra.run.smithi051.stdout: "endpoints": [ 2020-11-09T18:18:20.508 INFO:teuthology.orchestra.run.smithi051.stdout: "http://smithi078.front.sepia.ceph.com:8000" 2020-11-09T18:18:20.508 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:18:20.508 INFO:teuthology.orchestra.run.smithi051.stdout: "log_meta": "false", 2020-11-09T18:18:20.508 INFO:teuthology.orchestra.run.smithi051.stdout: "log_data": "true", 2020-11-09T18:18:20.515 INFO:teuthology.orchestra.run.smithi051.stdout: "bucket_index_max_shards": 11, 2020-11-09T18:18:20.515 INFO:teuthology.orchestra.run.smithi051.stdout: "read_only": "false", 2020-11-09T18:18:20.516 INFO:teuthology.orchestra.run.smithi051.stdout: "tier_type": "", 2020-11-09T18:18:20.516 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from_all": "false", 2020-11-09T18:18:20.516 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from": [], 2020-11-09T18:18:20.516 INFO:teuthology.orchestra.run.smithi051.stdout: "redirect_zone": "" 2020-11-09T18:18:20.516 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:18:20.516 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:20.517 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6", 2020-11-09T18:18:20.517 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "test-zone3", 2020-11-09T18:18:20.517 INFO:teuthology.orchestra.run.smithi051.stdout: "endpoints": [ 2020-11-09T18:18:20.517 INFO:teuthology.orchestra.run.smithi051.stdout: "http://smithi051.front.sepia.ceph.com:8001" 2020-11-09T18:18:20.517 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:18:20.518 INFO:teuthology.orchestra.run.smithi051.stdout: "log_meta": "false", 2020-11-09T18:18:20.518 INFO:teuthology.orchestra.run.smithi051.stdout: "log_data": "true", 2020-11-09T18:18:20.518 INFO:teuthology.orchestra.run.smithi051.stdout: "bucket_index_max_shards": 11, 2020-11-09T18:18:20.518 INFO:teuthology.orchestra.run.smithi051.stdout: "read_only": "false", 2020-11-09T18:18:20.518 INFO:teuthology.orchestra.run.smithi051.stdout: "tier_type": "", 2020-11-09T18:18:20.519 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from_all": "true", 2020-11-09T18:18:20.519 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from": [], 2020-11-09T18:18:20.519 INFO:teuthology.orchestra.run.smithi051.stdout: "redirect_zone": "" 2020-11-09T18:18:20.519 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:18:20.519 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:18:20.520 INFO:teuthology.orchestra.run.smithi051.stdout: "placement_targets": [ 2020-11-09T18:18:20.520 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:20.520 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "default-placement", 2020-11-09T18:18:20.520 INFO:teuthology.orchestra.run.smithi051.stdout: "tags": [], 2020-11-09T18:18:20.520 INFO:teuthology.orchestra.run.smithi051.stdout: "storage_classes": [ 2020-11-09T18:18:20.521 INFO:teuthology.orchestra.run.smithi051.stdout: "STANDARD" 2020-11-09T18:18:20.521 INFO:teuthology.orchestra.run.smithi051.stdout: ] 2020-11-09T18:18:20.521 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:18:20.521 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:18:20.522 INFO:teuthology.orchestra.run.smithi051.stdout: "default_placement": "default-placement", 2020-11-09T18:18:20.522 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_id": "0de9c5fb-e243-4abd-b414-59ffbc2cb575", 2020-11-09T18:18:20.522 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_policy": { 2020-11-09T18:18:20.522 INFO:teuthology.orchestra.run.smithi051.stdout: "groups": [] 2020-11-09T18:18:20.522 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:18:20.523 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:18:20.523 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:18:20.523 INFO:teuthology.orchestra.run.smithi051.stdout: "short_zone_ids": [ 2020-11-09T18:18:20.523 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:20.523 INFO:teuthology.orchestra.run.smithi051.stdout: "key": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:18:20.523 INFO:teuthology.orchestra.run.smithi051.stdout: "val": 2019612975 2020-11-09T18:18:20.524 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:18:20.524 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:20.525 INFO:teuthology.orchestra.run.smithi051.stdout: "key": "4600c141-233a-40b8-a0db-e3e3a34df6c3", 2020-11-09T18:18:20.525 INFO:teuthology.orchestra.run.smithi051.stdout: "val": 1720005997 2020-11-09T18:18:20.526 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:18:20.526 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:20.526 INFO:teuthology.orchestra.run.smithi051.stdout: "key": "4d0e43a6-0ddb-4d76-979c-891a9abfd6b4", 2020-11-09T18:18:20.526 INFO:teuthology.orchestra.run.smithi051.stdout: "val": 1250118036 2020-11-09T18:18:20.526 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:18:20.527 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:20.527 INFO:teuthology.orchestra.run.smithi051.stdout: "key": "4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc", 2020-11-09T18:18:20.527 INFO:teuthology.orchestra.run.smithi051.stdout: "val": 2281443694 2020-11-09T18:18:20.527 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:18:20.527 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:20.527 INFO:teuthology.orchestra.run.smithi051.stdout: "key": "a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6", 2020-11-09T18:18:20.527 INFO:teuthology.orchestra.run.smithi051.stdout: "val": 648291153 2020-11-09T18:18:20.528 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:18:20.528 INFO:teuthology.orchestra.run.smithi051.stdout: ] 2020-11-09T18:18:20.528 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:18:20.528 INFO:teuthology.orchestra.run.smithi051.stdout: "master_zonegroup": "test-zonegroup", 2020-11-09T18:18:20.528 INFO:teuthology.orchestra.run.smithi051.stdout: "master_zone": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:18:20.528 INFO:teuthology.orchestra.run.smithi051.stdout: "period_config": { 2020-11-09T18:18:20.528 INFO:teuthology.orchestra.run.smithi051.stdout: "bucket_quota": { 2020-11-09T18:18:20.529 INFO:teuthology.orchestra.run.smithi051.stdout: "enabled": false, 2020-11-09T18:18:20.529 INFO:teuthology.orchestra.run.smithi051.stdout: "check_on_raw": false, 2020-11-09T18:18:20.529 INFO:teuthology.orchestra.run.smithi051.stdout: "max_size": -1, 2020-11-09T18:18:20.529 INFO:teuthology.orchestra.run.smithi051.stdout: "max_size_kb": 0, 2020-11-09T18:18:20.529 INFO:teuthology.orchestra.run.smithi051.stdout: "max_objects": -1 2020-11-09T18:18:20.529 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:18:20.529 INFO:teuthology.orchestra.run.smithi051.stdout: "user_quota": { 2020-11-09T18:18:20.530 INFO:teuthology.orchestra.run.smithi051.stdout: "enabled": false, 2020-11-09T18:18:20.530 INFO:teuthology.orchestra.run.smithi051.stdout: "check_on_raw": false, 2020-11-09T18:18:20.530 INFO:teuthology.orchestra.run.smithi051.stdout: "max_size": -1, 2020-11-09T18:18:20.531 INFO:teuthology.orchestra.run.smithi051.stdout: "max_size_kb": 0, 2020-11-09T18:18:20.531 INFO:teuthology.orchestra.run.smithi051.stdout: "max_objects": -1 2020-11-09T18:18:20.531 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:18:20.531 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:18:20.531 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_id": "0de9c5fb-e243-4abd-b414-59ffbc2cb575", 2020-11-09T18:18:20.531 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_name": "test-realm", 2020-11-09T18:18:20.531 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 4 2020-11-09T18:18:20.532 INFO:teuthology.orchestra.run.smithi051.stdout:} 2020-11-09T18:18:20.533 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:18:20.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:18:20.534 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-09T18:18:20.614 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:18:20.615 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:18:20.666 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:18:20.661+0000 7f550ab890c0 0 zone id 4d0e43a6-0ddb-4d76-979c-891a9abfd6b4 is not a part of zonegroup test-zonegroup 2020-11-09T18:18:20.667 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:18:20.661+0000 7f550ab890c0 0 zone id 4d0e43a6-0ddb-4d76-979c-891a9abfd6b4 is not a part of zonegroup default 2020-11-09T18:18:20.688 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:18:20.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:18:20.690 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-09T18:18:20.773 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:18:20.773 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:18:20.831 INFO:teuthology.orchestra.run.smithi051.stderr:2020-11-09T18:18:20.825+0000 7f2ce59960c0 0 ERROR: zonegroup remove_zg has a non existent master zone 2020-11-09T18:18:20.840 DEBUG:teuthology.orchestra.run:got remote process result: 22 2020-11-09T18:18:20.840 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:18:20.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:18:20.841 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-09T18:18:20.922 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:18:20.922 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:18:20.957 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:18:20.958 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:18:20.959 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-09T18:18:21.047 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:18:21.047 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:18:21.405 INFO:teuthology.orchestra.run.smithi051.stdout:{ 2020-11-09T18:18:21.406 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:18:21.406 INFO:teuthology.orchestra.run.smithi051.stdout: "epoch": 6, 2020-11-09T18:18:21.406 INFO:teuthology.orchestra.run.smithi051.stdout: "predecessor_uuid": "bbf9f070-21b9-4940-95e9-d031ba1c647f", 2020-11-09T18:18:21.406 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_status": [ 2020-11-09T18:18:21.406 INFO:teuthology.orchestra.run.smithi051.stdout: "1_1604945680.637050_159.1", 2020-11-09T18:18:21.406 INFO:teuthology.orchestra.run.smithi051.stdout: "", 2020-11-09T18:18:21.407 INFO:teuthology.orchestra.run.smithi051.stdout: "", 2020-11-09T18:18:21.407 INFO:teuthology.orchestra.run.smithi051.stdout: "" 2020-11-09T18:18:21.407 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:18:21.407 INFO:teuthology.orchestra.run.smithi051.stdout: "period_map": { 2020-11-09T18:18:21.407 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:18:21.407 INFO:teuthology.orchestra.run.smithi051.stdout: "zonegroups": [ 2020-11-09T18:18:21.407 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:21.408 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "test-zonegroup", 2020-11-09T18:18:21.408 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "test-zonegroup", 2020-11-09T18:18:21.408 INFO:teuthology.orchestra.run.smithi051.stdout: "api_name": "", 2020-11-09T18:18:21.408 INFO:teuthology.orchestra.run.smithi051.stdout: "is_master": "true", 2020-11-09T18:18:21.408 INFO:teuthology.orchestra.run.smithi051.stdout: "endpoints": [ 2020-11-09T18:18:21.408 INFO:teuthology.orchestra.run.smithi051.stdout: "http://smithi051.front.sepia.ceph.com:8000" 2020-11-09T18:18:21.408 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:18:21.409 INFO:teuthology.orchestra.run.smithi051.stdout: "hostnames": [], 2020-11-09T18:18:21.409 INFO:teuthology.orchestra.run.smithi051.stdout: "hostnames_s3website": [], 2020-11-09T18:18:21.409 INFO:teuthology.orchestra.run.smithi051.stdout: "master_zone": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:18:21.409 INFO:teuthology.orchestra.run.smithi051.stdout: "zones": [ 2020-11-09T18:18:21.409 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:21.409 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:18:21.409 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "test-zone1", 2020-11-09T18:18:21.410 INFO:teuthology.orchestra.run.smithi051.stdout: "endpoints": [ 2020-11-09T18:18:21.410 INFO:teuthology.orchestra.run.smithi051.stdout: "http://smithi051.front.sepia.ceph.com:8000" 2020-11-09T18:18:21.410 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:18:21.410 INFO:teuthology.orchestra.run.smithi051.stdout: "log_meta": "false", 2020-11-09T18:18:21.410 INFO:teuthology.orchestra.run.smithi051.stdout: "log_data": "true", 2020-11-09T18:18:21.410 INFO:teuthology.orchestra.run.smithi051.stdout: "bucket_index_max_shards": 11, 2020-11-09T18:18:21.410 INFO:teuthology.orchestra.run.smithi051.stdout: "read_only": "false", 2020-11-09T18:18:21.411 INFO:teuthology.orchestra.run.smithi051.stdout: "tier_type": "", 2020-11-09T18:18:21.411 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from_all": "true", 2020-11-09T18:18:21.411 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from": [], 2020-11-09T18:18:21.411 INFO:teuthology.orchestra.run.smithi051.stdout: "redirect_zone": "" 2020-11-09T18:18:21.411 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:18:21.411 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:21.411 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "4600c141-233a-40b8-a0db-e3e3a34df6c3", 2020-11-09T18:18:21.411 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "test-zone4", 2020-11-09T18:18:21.412 INFO:teuthology.orchestra.run.smithi051.stdout: "endpoints": [ 2020-11-09T18:18:21.412 INFO:teuthology.orchestra.run.smithi051.stdout: "http://smithi078.front.sepia.ceph.com:8001" 2020-11-09T18:18:21.412 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:18:21.412 INFO:teuthology.orchestra.run.smithi051.stdout: "log_meta": "false", 2020-11-09T18:18:21.412 INFO:teuthology.orchestra.run.smithi051.stdout: "log_data": "true", 2020-11-09T18:18:21.412 INFO:teuthology.orchestra.run.smithi051.stdout: "bucket_index_max_shards": 11, 2020-11-09T18:18:21.412 INFO:teuthology.orchestra.run.smithi051.stdout: "read_only": "false", 2020-11-09T18:18:21.413 INFO:teuthology.orchestra.run.smithi051.stdout: "tier_type": "pubsub", 2020-11-09T18:18:21.413 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from_all": "false", 2020-11-09T18:18:21.413 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from": [ 2020-11-09T18:18:21.413 INFO:teuthology.orchestra.run.smithi051.stdout: "test-zone1" 2020-11-09T18:18:21.413 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:18:21.413 INFO:teuthology.orchestra.run.smithi051.stdout: "redirect_zone": "" 2020-11-09T18:18:21.413 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:18:21.414 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:21.414 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc", 2020-11-09T18:18:21.414 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "test-zone2", 2020-11-09T18:18:21.414 INFO:teuthology.orchestra.run.smithi051.stdout: "endpoints": [ 2020-11-09T18:18:21.414 INFO:teuthology.orchestra.run.smithi051.stdout: "http://smithi078.front.sepia.ceph.com:8000" 2020-11-09T18:18:21.414 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:18:21.414 INFO:teuthology.orchestra.run.smithi051.stdout: "log_meta": "false", 2020-11-09T18:18:21.415 INFO:teuthology.orchestra.run.smithi051.stdout: "log_data": "true", 2020-11-09T18:18:21.415 INFO:teuthology.orchestra.run.smithi051.stdout: "bucket_index_max_shards": 11, 2020-11-09T18:18:21.415 INFO:teuthology.orchestra.run.smithi051.stdout: "read_only": "false", 2020-11-09T18:18:21.415 INFO:teuthology.orchestra.run.smithi051.stdout: "tier_type": "", 2020-11-09T18:18:21.415 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from_all": "false", 2020-11-09T18:18:21.415 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from": [], 2020-11-09T18:18:21.415 INFO:teuthology.orchestra.run.smithi051.stdout: "redirect_zone": "" 2020-11-09T18:18:21.415 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:18:21.416 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:21.416 INFO:teuthology.orchestra.run.smithi051.stdout: "id": "a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6", 2020-11-09T18:18:21.416 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "test-zone3", 2020-11-09T18:18:21.416 INFO:teuthology.orchestra.run.smithi051.stdout: "endpoints": [ 2020-11-09T18:18:21.416 INFO:teuthology.orchestra.run.smithi051.stdout: "http://smithi051.front.sepia.ceph.com:8001" 2020-11-09T18:18:21.416 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:18:21.416 INFO:teuthology.orchestra.run.smithi051.stdout: "log_meta": "false", 2020-11-09T18:18:21.417 INFO:teuthology.orchestra.run.smithi051.stdout: "log_data": "true", 2020-11-09T18:18:21.417 INFO:teuthology.orchestra.run.smithi051.stdout: "bucket_index_max_shards": 11, 2020-11-09T18:18:21.417 INFO:teuthology.orchestra.run.smithi051.stdout: "read_only": "false", 2020-11-09T18:18:21.417 INFO:teuthology.orchestra.run.smithi051.stdout: "tier_type": "", 2020-11-09T18:18:21.417 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from_all": "true", 2020-11-09T18:18:21.417 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_from": [], 2020-11-09T18:18:21.417 INFO:teuthology.orchestra.run.smithi051.stdout: "redirect_zone": "" 2020-11-09T18:18:21.418 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:18:21.418 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:18:21.418 INFO:teuthology.orchestra.run.smithi051.stdout: "placement_targets": [ 2020-11-09T18:18:21.418 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:21.418 INFO:teuthology.orchestra.run.smithi051.stdout: "name": "default-placement", 2020-11-09T18:18:21.418 INFO:teuthology.orchestra.run.smithi051.stdout: "tags": [], 2020-11-09T18:18:21.418 INFO:teuthology.orchestra.run.smithi051.stdout: "storage_classes": [ 2020-11-09T18:18:21.419 INFO:teuthology.orchestra.run.smithi051.stdout: "STANDARD" 2020-11-09T18:18:21.419 INFO:teuthology.orchestra.run.smithi051.stdout: ] 2020-11-09T18:18:21.419 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:18:21.419 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:18:21.419 INFO:teuthology.orchestra.run.smithi051.stdout: "default_placement": "default-placement", 2020-11-09T18:18:21.419 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_id": "0de9c5fb-e243-4abd-b414-59ffbc2cb575", 2020-11-09T18:18:21.419 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_policy": { 2020-11-09T18:18:21.420 INFO:teuthology.orchestra.run.smithi051.stdout: "groups": [] 2020-11-09T18:18:21.420 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:18:21.420 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:18:21.420 INFO:teuthology.orchestra.run.smithi051.stdout: ], 2020-11-09T18:18:21.441 INFO:teuthology.orchestra.run.smithi051.stdout: "short_zone_ids": [ 2020-11-09T18:18:21.441 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:21.442 INFO:teuthology.orchestra.run.smithi051.stdout: "key": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:18:21.442 INFO:teuthology.orchestra.run.smithi051.stdout: "val": 2019612975 2020-11-09T18:18:21.442 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:18:21.442 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:21.442 INFO:teuthology.orchestra.run.smithi051.stdout: "key": "4600c141-233a-40b8-a0db-e3e3a34df6c3", 2020-11-09T18:18:21.442 INFO:teuthology.orchestra.run.smithi051.stdout: "val": 1720005997 2020-11-09T18:18:21.443 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:18:21.443 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:21.443 INFO:teuthology.orchestra.run.smithi051.stdout: "key": "4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc", 2020-11-09T18:18:21.443 INFO:teuthology.orchestra.run.smithi051.stdout: "val": 2281443694 2020-11-09T18:18:21.443 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:18:21.443 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:21.443 INFO:teuthology.orchestra.run.smithi051.stdout: "key": "a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6", 2020-11-09T18:18:21.444 INFO:teuthology.orchestra.run.smithi051.stdout: "val": 648291153 2020-11-09T18:18:21.444 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:18:21.444 INFO:teuthology.orchestra.run.smithi051.stdout: ] 2020-11-09T18:18:21.444 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:18:21.444 INFO:teuthology.orchestra.run.smithi051.stdout: "master_zonegroup": "test-zonegroup", 2020-11-09T18:18:21.444 INFO:teuthology.orchestra.run.smithi051.stdout: "master_zone": "11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:18:21.444 INFO:teuthology.orchestra.run.smithi051.stdout: "period_config": { 2020-11-09T18:18:21.445 INFO:teuthology.orchestra.run.smithi051.stdout: "bucket_quota": { 2020-11-09T18:18:21.445 INFO:teuthology.orchestra.run.smithi051.stdout: "enabled": false, 2020-11-09T18:18:21.445 INFO:teuthology.orchestra.run.smithi051.stdout: "check_on_raw": false, 2020-11-09T18:18:21.445 INFO:teuthology.orchestra.run.smithi051.stdout: "max_size": -1, 2020-11-09T18:18:21.445 INFO:teuthology.orchestra.run.smithi051.stdout: "max_size_kb": 0, 2020-11-09T18:18:21.445 INFO:teuthology.orchestra.run.smithi051.stdout: "max_objects": -1 2020-11-09T18:18:21.445 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:18:21.445 INFO:teuthology.orchestra.run.smithi051.stdout: "user_quota": { 2020-11-09T18:18:21.446 INFO:teuthology.orchestra.run.smithi051.stdout: "enabled": false, 2020-11-09T18:18:21.446 INFO:teuthology.orchestra.run.smithi051.stdout: "check_on_raw": false, 2020-11-09T18:18:21.446 INFO:teuthology.orchestra.run.smithi051.stdout: "max_size": -1, 2020-11-09T18:18:21.446 INFO:teuthology.orchestra.run.smithi051.stdout: "max_size_kb": 0, 2020-11-09T18:18:21.446 INFO:teuthology.orchestra.run.smithi051.stdout: "max_objects": -1 2020-11-09T18:18:21.446 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:18:21.446 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:18:21.447 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_id": "0de9c5fb-e243-4abd-b414-59ffbc2cb575", 2020-11-09T18:18:21.447 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_name": "test-realm", 2020-11-09T18:18:21.447 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 4 2020-11-09T18:18:21.447 INFO:teuthology.orchestra.run.smithi051.stdout:} 2020-11-09T18:18:21.448 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_zg_master_zone_delete ... ok 2020-11-09T18:18:21.449 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ymzeui-50 2020-11-09T18:18:22.098 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ymzeui-51 2020-11-09T18:18:22.188 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ymzeui-52 2020-11-09T18:18:22.400 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ymzeui-53 2020-11-09T18:18:22.454 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ymzeui-54 2020-11-09T18:18:22.550 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ymzeui-55 2020-11-09T18:18:25.834 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_set_bucket_website ... tasks.rgw_multi.tests.test_set_bucket_policy ... ok 2020-11-09T18:18:25.835 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ymzeui-56 2020-11-09T18:18:26.673 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ymzeui-57 2020-11-09T18:18:27.452 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ymzeui-58 2020-11-09T18:18:28.172 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:18:28.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:18:28.173 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:18:28.210 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:18:28.211 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:18:28.458 INFO:teuthology.orchestra.run.smithi051.stderr:No --period given, using current period=f6258aba-300a-4d5d-ad93-a7c198a0e0a1 2020-11-09T18:18:28.491 INFO:teuthology.orchestra.run.smithi051.stdout:[ 2020-11-09T18:18:28.492 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:28.493 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945908.145521_359.1", 2020-11-09T18:18:28.493 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "2020-11-09T18:18:28.145521Z" 2020-11-09T18:18:28.493 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:18:28.493 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:28.493 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:18:28.493 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:18:28.494 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:18:28.494 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:28.494 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:18:28.494 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:18:28.494 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:18:28.494 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:18:28.494 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:18:28.495 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:18:28.495 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:18:28.495 INFO:teuthology.orchestra.run.smithi051.stdout:] 2020-11-09T18:18:28.496 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-09T18:18:28.496 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:18:28.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:18:28.498 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:18:28.541 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:18:28.542 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:18:28.782 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:18:28.783 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:18:28.783 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:18:28.783 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:18:28.783 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:18:28.784 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:18:28.784 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:18:28.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:28.784 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:18:28.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:28.785 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:18:28.785 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:18:28.785 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:18:28.785 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945825.625804_254.1", 2020-11-09T18:18:28.786 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:18:28.786 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:18:28.786 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:18:28.786 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:17:05.625804Z", 2020-11-09T18:18:28.786 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:18:28.786 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:28.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:28.787 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:28.787 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:18:28.787 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:18:28.787 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:18:28.788 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:18:28.788 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:18:28.788 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:18:28.788 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:18:28.789 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:18:28.789 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:18:28.789 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:28.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:28.789 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:28.790 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:18:28.790 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:18:28.790 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:18:28.790 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:18:28.790 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:18:28.791 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:18:28.791 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:18:28.791 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:18:28.791 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:18:28.791 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:28.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:28.792 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:28.792 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:18:28.792 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:18:28.792 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:18:28.793 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:18:28.793 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:18:28.793 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:18:28.793 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:18:28.793 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:18:28.794 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:18:28.794 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:28.794 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:28.794 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:18:28.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:28.795 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:18:28.795 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:18:28.795 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:18:28.795 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:28.795 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:18:28.797 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945908.145521_359.1 target=1_1604945825.625804_254.1 2020-11-09T18:18:33.798 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:18:33.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:18:33.799 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:18:33.841 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:18:33.842 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:18:34.108 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:18:34.108 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:18:34.108 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:18:34.109 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:18:34.109 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:18:34.109 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:18:34.109 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:18:34.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:34.109 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:18:34.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:34.110 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:18:34.110 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:18:34.110 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:18:34.110 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945825.625804_254.1", 2020-11-09T18:18:34.110 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:18:34.110 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:18:34.110 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:18:34.111 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:17:05.625804Z", 2020-11-09T18:18:34.111 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:18:34.111 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:34.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:34.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:34.111 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:18:34.111 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:18:34.112 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:18:34.112 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:18:34.112 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:18:34.112 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:18:34.112 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:18:34.112 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:18:34.112 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:18:34.113 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:34.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:34.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:34.113 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:18:34.113 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:18:34.113 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:18:34.113 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:18:34.114 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:18:34.114 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:18:34.114 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:18:34.114 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:18:34.114 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:18:34.114 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:34.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:34.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:34.115 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:18:34.115 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:18:34.115 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:18:34.115 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:18:34.115 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:18:34.115 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:18:34.115 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:18:34.116 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:18:34.116 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:18:34.116 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:34.116 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:34.116 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:18:34.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:34.116 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:18:34.117 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:18:34.117 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:18:34.117 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:34.117 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:18:34.118 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945908.145521_359.1 target=1_1604945825.625804_254.1 2020-11-09T18:18:39.119 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:18:39.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:18:39.121 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:18:39.163 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:18:39.164 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:18:39.424 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:18:39.424 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:18:39.425 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:18:39.425 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:18:39.425 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:18:39.425 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:18:39.425 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:18:39.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:39.425 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:18:39.426 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:39.426 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:18:39.426 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:18:39.426 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:18:39.426 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945825.625804_254.1", 2020-11-09T18:18:39.426 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:18:39.426 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:18:39.427 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:18:39.427 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:17:05.625804Z", 2020-11-09T18:18:39.427 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:18:39.427 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:39.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:39.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:39.428 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:18:39.428 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:18:39.428 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:18:39.428 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:18:39.429 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:18:39.429 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:18:39.429 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:18:39.429 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:18:39.429 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:18:39.429 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:39.430 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:39.430 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:39.430 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:18:39.430 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:18:39.430 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:18:39.430 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:18:39.431 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:18:39.431 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:18:39.431 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:18:39.431 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:18:39.431 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:18:39.431 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:39.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:39.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:39.432 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:18:39.432 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:18:39.432 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:18:39.432 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:18:39.433 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:18:39.433 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:18:39.433 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:18:39.433 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:18:39.433 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:18:39.433 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:39.433 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:39.434 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:18:39.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:39.434 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:18:39.434 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:18:39.434 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:18:39.434 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:39.435 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:18:39.436 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945908.145521_359.1 target=1_1604945825.625804_254.1 2020-11-09T18:18:44.437 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:18:44.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:18:44.438 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:18:44.480 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:18:44.480 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:18:44.736 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:18:44.736 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:18:44.736 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:18:44.736 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:18:44.737 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:18:44.737 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:18:44.737 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:18:44.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:44.737 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:18:44.737 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:44.737 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:18:44.738 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:18:44.738 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:18:44.738 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945825.625804_254.1", 2020-11-09T18:18:44.738 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:18:44.738 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:18:44.738 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:18:44.739 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:17:05.625804Z", 2020-11-09T18:18:44.739 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:18:44.739 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:44.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:44.739 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:44.739 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:18:44.739 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:18:44.739 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:18:44.740 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:18:44.740 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:18:44.740 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:18:44.740 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:18:44.740 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:18:44.740 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:18:44.740 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:44.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:44.741 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:44.741 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:18:44.741 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:18:44.741 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:18:44.741 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:18:44.741 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:18:44.742 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:18:44.742 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:18:44.742 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:18:44.742 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:18:44.742 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:44.742 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:44.742 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:44.743 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:18:44.743 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:18:44.743 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:18:44.743 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:18:44.743 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:18:44.743 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:18:44.743 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:18:44.743 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:18:44.744 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:18:44.744 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:44.744 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:44.745 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:18:44.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:44.745 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:18:44.745 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:18:44.745 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:18:44.745 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:44.746 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:18:44.747 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945908.145521_359.1 target=1_1604945825.625804_254.1 2020-11-09T18:18:49.748 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:18:49.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:18:49.750 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:18:49.791 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:18:49.791 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:18:50.061 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:18:50.062 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:18:50.062 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:18:50.062 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:18:50.062 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:18:50.062 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:18:50.062 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:18:50.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:50.063 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:18:50.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:50.063 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:18:50.063 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:18:50.063 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:18:50.063 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945825.625804_254.1", 2020-11-09T18:18:50.064 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:18:50.064 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:18:50.064 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:18:50.064 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:17:05.625804Z", 2020-11-09T18:18:50.064 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:18:50.064 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:50.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:50.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:50.065 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:18:50.065 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:18:50.065 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:18:50.065 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:18:50.065 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:18:50.065 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:18:50.066 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:18:50.066 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:18:50.066 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:18:50.066 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:50.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:50.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:50.066 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:18:50.066 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:18:50.067 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:18:50.067 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:18:50.067 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:18:50.067 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:18:50.067 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:18:50.067 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:18:50.067 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:18:50.068 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:50.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:50.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:50.068 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:18:50.068 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:18:50.068 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:18:50.068 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:18:50.069 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:18:50.069 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:18:50.069 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:18:50.069 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:18:50.069 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:18:50.069 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:50.069 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:50.069 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:18:50.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:50.070 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:18:50.070 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:18:50.070 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:18:50.070 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:50.070 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:18:50.071 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945908.145521_359.1 target=1_1604945825.625804_254.1 2020-11-09T18:18:55.073 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:18:55.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:18:55.074 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:18:55.115 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:18:55.115 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:18:55.381 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:18:55.381 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:18:55.381 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:18:55.382 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:18:55.382 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:18:55.382 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:18:55.382 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:18:55.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:55.382 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:18:55.382 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:55.383 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:18:55.383 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:18:55.383 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:18:55.383 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945825.625804_254.1", 2020-11-09T18:18:55.383 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:18:55.383 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:18:55.383 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:18:55.384 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:17:05.625804Z", 2020-11-09T18:18:55.384 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:18:55.384 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:55.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:55.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:55.384 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:18:55.384 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:18:55.385 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:18:55.385 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:18:55.385 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:18:55.385 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:18:55.385 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:18:55.385 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:18:55.385 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:18:55.386 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:55.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:55.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:55.386 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:18:55.386 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:18:55.386 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:18:55.386 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:18:55.386 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:18:55.387 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:18:55.387 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:18:55.387 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:18:55.387 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:18:55.387 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:55.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:55.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:18:55.388 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:18:55.388 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:18:55.388 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:18:55.388 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:18:55.388 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:18:55.388 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:18:55.388 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:18:55.389 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:18:55.389 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:18:55.389 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:55.389 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:55.389 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:18:55.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:18:55.389 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:18:55.390 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:18:55.390 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:18:55.390 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:18:55.390 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:18:55.391 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945908.145521_359.1 target=1_1604945825.625804_254.1 2020-11-09T18:19:00.393 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:19:00.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:19:00.394 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:19:00.434 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:19:00.435 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:19:00.697 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:19:00.698 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:19:00.698 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:19:00.698 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:19:00.699 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:19:00.699 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:19:00.699 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:19:00.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:00.699 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:19:00.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:00.700 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:19:00.700 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:00.700 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:00.700 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945825.625804_254.1", 2020-11-09T18:19:00.700 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:00.700 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:19:00.701 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:00.701 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:17:05.625804Z", 2020-11-09T18:19:00.701 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:19:00.701 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:00.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:00.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:00.702 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:19:00.702 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:00.702 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:00.702 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:19:00.702 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:00.703 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:19:00.703 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:00.703 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:19:00.703 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:19:00.703 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:00.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:00.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:00.704 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:19:00.704 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:00.704 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:00.704 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:19:00.705 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:00.705 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:19:00.705 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:00.705 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:19:00.705 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:19:00.705 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:00.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:00.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:00.706 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:19:00.706 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:00.706 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:00.706 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:19:00.707 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:00.707 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:19:00.707 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:00.707 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:19:00.707 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:19:00.707 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:00.707 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:00.708 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:19:00.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:00.708 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:19:00.708 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:19:00.708 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:19:00.708 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:00.709 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:19:00.710 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945908.145521_359.1 target=1_1604945825.625804_254.1 2020-11-09T18:19:05.711 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:19:05.712 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:19:05.713 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:19:05.752 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:19:05.752 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:19:06.005 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:19:06.005 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:19:06.006 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:19:06.006 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:19:06.006 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:19:06.006 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:19:06.006 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:19:06.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:06.007 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:19:06.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:06.007 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:19:06.007 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:06.007 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:06.007 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945825.625804_254.1", 2020-11-09T18:19:06.007 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:06.008 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:19:06.008 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:06.008 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:17:05.625804Z", 2020-11-09T18:19:06.008 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:19:06.008 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:06.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:06.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:06.009 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:19:06.009 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:06.009 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:06.009 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:19:06.009 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:06.009 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:19:06.009 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:06.009 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:19:06.010 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:19:06.010 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:06.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:06.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:06.010 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:19:06.010 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:06.010 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:06.014 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:19:06.014 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:06.014 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:19:06.015 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:06.015 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:19:06.015 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:19:06.015 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:06.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:06.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:06.016 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:19:06.016 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:06.016 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:06.016 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:19:06.016 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:06.016 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:19:06.016 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:06.017 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:19:06.017 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:19:06.017 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:06.017 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:06.017 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:19:06.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:06.018 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:19:06.018 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:19:06.018 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:19:06.018 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:06.018 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:19:06.019 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945908.145521_359.1 target=1_1604945825.625804_254.1 2020-11-09T18:19:11.021 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:19:11.022 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:19:11.023 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:19:11.064 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:19:11.064 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:19:11.336 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:19:11.337 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:19:11.337 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:19:11.337 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:19:11.337 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:19:11.337 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:19:11.338 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:19:11.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:11.338 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:19:11.338 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:11.338 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:19:11.338 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:11.339 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:11.339 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945825.625804_254.1", 2020-11-09T18:19:11.339 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:11.339 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:19:11.339 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:11.339 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:17:05.625804Z", 2020-11-09T18:19:11.340 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:19:11.340 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:11.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:11.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:11.340 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:19:11.341 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:11.341 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:11.341 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:19:11.343 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:11.343 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:19:11.343 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:11.343 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:19:11.343 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:19:11.344 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:11.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:11.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:11.344 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:19:11.344 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:11.344 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:11.344 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:19:11.345 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:11.345 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:19:11.345 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:11.345 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:19:11.345 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:19:11.345 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:11.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:11.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:11.346 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:19:11.346 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:11.346 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:11.346 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:19:11.347 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:11.347 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:19:11.347 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:11.347 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:19:11.347 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:19:11.347 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:11.347 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:11.348 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:19:11.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:11.348 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:19:11.348 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:19:11.348 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:19:11.348 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:11.348 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:19:11.350 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945908.145521_359.1 target=1_1604945825.625804_254.1 2020-11-09T18:19:16.351 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:19:16.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:19:16.425 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:19:16.467 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:19:16.467 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:19:16.713 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:19:16.713 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:19:16.713 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:19:16.714 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:19:16.714 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:19:16.714 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:19:16.714 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:19:16.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:16.714 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:19:16.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:16.715 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:19:16.715 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:16.715 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:16.715 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945825.625804_254.1", 2020-11-09T18:19:16.715 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:16.715 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:19:16.715 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:16.716 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:17:05.625804Z", 2020-11-09T18:19:16.716 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:19:16.716 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:16.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:16.716 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:16.716 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:19:16.716 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:16.717 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:16.717 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:19:16.717 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:16.717 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:19:16.717 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:16.717 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:19:16.717 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:19:16.718 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:16.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:16.718 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:16.718 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:19:16.718 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:16.718 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:16.718 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:19:16.719 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:16.719 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:19:16.719 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:16.719 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:19:16.719 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:19:16.719 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:16.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:16.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:16.720 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:19:16.720 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:16.720 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:16.720 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:19:16.720 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:16.720 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:19:16.720 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:16.721 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:19:16.721 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:19:16.721 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:16.721 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:16.721 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:19:16.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:16.721 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:19:16.721 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:19:16.722 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:19:16.722 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:16.722 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:19:16.723 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945908.145521_359.1 target=1_1604945825.625804_254.1 2020-11-09T18:19:21.724 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:19:21.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:19:21.726 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:19:21.767 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:19:21.768 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:19:22.048 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:19:22.048 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:19:22.048 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:19:22.049 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:19:22.049 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:19:22.049 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:19:22.049 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:19:22.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:22.049 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:19:22.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:22.050 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:19:22.050 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:22.050 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:22.050 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945825.625804_254.1", 2020-11-09T18:19:22.050 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:22.050 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:19:22.050 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:22.051 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:17:05.625804Z", 2020-11-09T18:19:22.051 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:19:22.051 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:22.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:22.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:22.051 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:19:22.051 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:22.052 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:22.052 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:19:22.052 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:22.052 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:19:22.052 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:22.052 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:19:22.052 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:19:22.053 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:22.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:22.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:22.053 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:19:22.053 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:22.053 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:22.053 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:19:22.054 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:22.054 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:19:22.054 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:22.054 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:19:22.054 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:19:22.054 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:22.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:22.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:22.055 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:19:22.055 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:22.055 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:22.055 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:19:22.055 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:22.055 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:19:22.055 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:22.056 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:19:22.056 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:19:22.056 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:22.056 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:22.056 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:19:22.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:22.056 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:19:22.057 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:19:22.057 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:19:22.057 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:22.057 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:19:22.058 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945908.145521_359.1 target=1_1604945825.625804_254.1 2020-11-09T18:19:27.060 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:19:27.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:19:27.061 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:19:27.101 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:19:27.101 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:19:27.380 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:19:27.381 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:19:27.381 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:19:27.381 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:19:27.381 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:19:27.381 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:19:27.382 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:19:27.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:27.382 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:19:27.382 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:27.382 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:19:27.382 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:27.382 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:27.383 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945825.625804_254.1", 2020-11-09T18:19:27.383 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:27.383 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:19:27.383 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:27.383 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:17:05.625804Z", 2020-11-09T18:19:27.383 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:19:27.383 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:27.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:27.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:27.384 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:19:27.384 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:27.384 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:27.384 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:19:27.384 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:27.385 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:19:27.385 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:27.385 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:19:27.385 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:19:27.385 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:27.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:27.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:27.385 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:19:27.386 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:27.386 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:27.386 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:19:27.386 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:27.386 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:19:27.386 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:27.386 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:19:27.387 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:19:27.387 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:27.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:27.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:27.387 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:19:27.387 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:27.387 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:27.388 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:19:27.388 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:27.388 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:19:27.388 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:27.388 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:19:27.388 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:19:27.388 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:27.388 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:27.389 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:19:27.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:27.389 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:19:27.389 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:19:27.389 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:19:27.389 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:27.389 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:19:27.390 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945908.145521_359.1 target=1_1604945825.625804_254.1 2020-11-09T18:19:32.392 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:19:32.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:19:32.393 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:19:32.434 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:19:32.434 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:19:32.684 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:19:32.685 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:19:32.685 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:19:32.685 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:19:32.685 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:19:32.686 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:19:32.686 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:19:32.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:32.686 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:19:32.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:32.686 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:19:32.687 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:32.687 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:32.687 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945825.625804_254.1", 2020-11-09T18:19:32.687 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:32.687 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:19:32.688 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:32.688 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:17:05.625804Z", 2020-11-09T18:19:32.688 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:19:32.688 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:32.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:32.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:32.689 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:19:32.689 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:32.689 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:32.689 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:19:32.689 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:32.690 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:19:32.690 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:32.690 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:19:32.690 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:19:32.690 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:32.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:32.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:32.691 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:19:32.691 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:32.691 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:32.691 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:19:32.691 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:32.691 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:19:32.691 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:32.692 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:19:32.692 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:19:32.692 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:32.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:32.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:32.692 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:19:32.692 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:32.693 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:32.693 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:19:32.693 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:32.693 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:19:32.693 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:32.693 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:19:32.693 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:19:32.693 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:32.694 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:32.694 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:19:32.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:32.694 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:19:32.694 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:19:32.694 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:19:32.695 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:32.695 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:19:32.696 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945908.145521_359.1 target=1_1604945825.625804_254.1 2020-11-09T18:19:37.697 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:19:37.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:19:37.699 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:19:37.739 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:19:37.739 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:19:38.004 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:19:38.004 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:19:38.004 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:19:38.004 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:19:38.005 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:19:38.005 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:19:38.005 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:19:38.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:38.005 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:19:38.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:38.005 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:19:38.006 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:38.006 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:38.006 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945825.625804_254.1", 2020-11-09T18:19:38.006 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:38.006 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:19:38.006 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:38.007 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:17:05.625804Z", 2020-11-09T18:19:38.007 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:19:38.007 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:38.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:38.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:38.007 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:19:38.007 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:38.008 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:38.008 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:19:38.008 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:38.008 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:19:38.008 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:38.008 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:19:38.008 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:19:38.008 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:38.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:38.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:38.009 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:19:38.009 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:38.009 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:38.009 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:19:38.009 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:38.010 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:19:38.010 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:38.010 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:19:38.010 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:19:38.010 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:38.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:38.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:38.011 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:19:38.011 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:38.011 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:38.011 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:19:38.011 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:38.011 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:19:38.011 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:38.011 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:19:38.012 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:19:38.012 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:38.012 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:38.012 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:19:38.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:38.012 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:19:38.012 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:19:38.013 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:19:38.013 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:38.013 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:19:38.014 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945908.145521_359.1 target=1_1604945825.625804_254.1 2020-11-09T18:19:43.015 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:19:43.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:19:43.016 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:19:43.055 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:19:43.056 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:19:43.329 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:19:43.329 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:19:43.329 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:19:43.330 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:19:43.330 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:19:43.330 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:19:43.330 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:19:43.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:43.330 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:19:43.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:43.331 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:19:43.331 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:43.331 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:43.331 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945825.625804_254.1", 2020-11-09T18:19:43.331 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:43.331 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:19:43.331 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:43.332 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:17:05.625804Z", 2020-11-09T18:19:43.332 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:19:43.332 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:43.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:43.332 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:43.332 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:19:43.332 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:43.333 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:43.333 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:19:43.333 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:43.333 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:19:43.333 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:43.333 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:19:43.333 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:19:43.334 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:43.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:43.334 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:43.334 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:19:43.334 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:43.334 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:43.334 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:19:43.335 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:43.335 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:19:43.335 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:43.335 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:19:43.335 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:19:43.335 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:43.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:43.335 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:43.336 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:19:43.336 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:43.336 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:43.336 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:19:43.336 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:43.336 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:19:43.336 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:43.337 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:19:43.337 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:19:43.337 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:43.337 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:43.337 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:19:43.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:43.337 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:19:43.338 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:19:43.338 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:19:43.338 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:43.338 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:19:43.339 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945908.145521_359.1 target=1_1604945825.625804_254.1 2020-11-09T18:19:48.340 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:19:48.342 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:19:48.342 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:19:48.384 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:19:48.384 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:19:48.651 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:19:49.146 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:19:49.147 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:19:49.147 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:19:49.147 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:19:49.147 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:19:49.147 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:19:49.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:49.148 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:19:49.148 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:49.148 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:19:49.148 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:49.148 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:49.148 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945825.625804_254.1", 2020-11-09T18:19:49.148 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:49.149 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:19:49.149 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:49.149 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:17:05.625804Z", 2020-11-09T18:19:49.149 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:19:49.149 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:49.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:49.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:49.150 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:19:49.150 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:49.150 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:49.150 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:19:49.150 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:49.150 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:19:49.151 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:49.151 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:19:49.151 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:19:49.151 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:49.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:49.151 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:49.151 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:19:49.152 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:49.152 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:49.152 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:19:49.152 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:49.152 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:19:49.152 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:49.152 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:19:49.153 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:19:49.153 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:49.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:49.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:49.153 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:19:49.153 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:49.153 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:49.154 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:19:49.154 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:49.154 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:19:49.154 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:49.154 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:19:49.155 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:19:49.155 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:49.155 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:49.155 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:19:49.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:49.155 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:19:49.155 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:19:49.155 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:19:49.156 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:49.156 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:19:49.157 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945908.145521_359.1 target=1_1604945825.625804_254.1 2020-11-09T18:19:54.158 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:19:54.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:19:54.159 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:19:54.199 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:19:54.199 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:19:54.461 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:19:54.462 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:19:54.462 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:19:54.462 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:19:54.462 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:19:54.463 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:19:54.463 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:19:54.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:54.463 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:19:54.463 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:54.463 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:19:54.463 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:54.464 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:54.464 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945825.625804_254.1", 2020-11-09T18:19:54.464 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:54.464 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:19:54.464 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:54.464 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:17:05.625804Z", 2020-11-09T18:19:54.464 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:19:54.465 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:54.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:54.465 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:54.465 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:19:54.465 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:54.465 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:54.465 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:19:54.465 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:54.466 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:19:54.466 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:54.466 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:19:54.466 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:19:54.466 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:54.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:54.467 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:54.467 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:19:54.467 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:54.467 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:54.467 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:19:54.467 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:54.468 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:19:54.468 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:54.468 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:19:54.468 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:19:54.468 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:54.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:54.468 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:54.468 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:19:54.469 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:54.469 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:54.469 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:19:54.469 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:54.469 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:19:54.469 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:54.469 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:19:54.470 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:19:54.470 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:54.470 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:54.470 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:19:54.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:54.470 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:19:54.470 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:19:54.471 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:19:54.471 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:54.471 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:19:54.472 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945908.145521_359.1 target=1_1604945825.625804_254.1 2020-11-09T18:19:59.473 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:19:59.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:19:59.475 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:19:59.515 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:19:59.515 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:19:59.761 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:19:59.762 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:19:59.762 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:19:59.762 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:19:59.762 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:19:59.762 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:19:59.763 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:19:59.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:59.763 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:19:59.763 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:59.763 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:19:59.763 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:59.763 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:59.764 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945825.625804_254.1", 2020-11-09T18:19:59.764 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:59.764 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:19:59.764 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:59.764 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:17:05.625804Z", 2020-11-09T18:19:59.764 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:19:59.764 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:59.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:59.765 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:59.765 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:19:59.765 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:59.765 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:59.765 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:19:59.765 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:59.766 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:19:59.766 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:59.766 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:19:59.766 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:19:59.767 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:59.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:59.767 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:59.768 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:19:59.768 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:59.768 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:59.768 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:19:59.769 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:59.769 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:19:59.769 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:59.769 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:19:59.770 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:19:59.770 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:59.770 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:59.771 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:19:59.771 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:19:59.771 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:19:59.771 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:19:59.772 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:19:59.772 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:19:59.772 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:19:59.773 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:19:59.773 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:19:59.773 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:19:59.773 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:59.774 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:59.774 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:19:59.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:19:59.775 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:19:59.775 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:19:59.775 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:19:59.775 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:19:59.776 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:19:59.777 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945908.145521_359.1 target=1_1604945825.625804_254.1 2020-11-09T18:20:04.778 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:20:04.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:20:04.780 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:20:04.823 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:20:04.824 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:20:05.093 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:20:05.093 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:20:05.093 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:20:05.094 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:20:05.094 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:20:05.094 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:20:05.094 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:20:05.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:05.094 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:20:05.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:05.095 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:20:05.095 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:05.095 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:20:05.095 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945825.625804_254.1", 2020-11-09T18:20:05.096 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:05.096 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:20:05.096 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:05.096 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:17:05.625804Z", 2020-11-09T18:20:05.096 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:20:05.096 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:05.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:05.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:05.097 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:20:05.097 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:05.097 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:20:05.097 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:20:05.097 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:05.097 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:20:05.098 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:05.098 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:20:05.098 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:20:05.098 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:05.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:05.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:05.098 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:20:05.099 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:05.099 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:20:05.099 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:20:05.099 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:05.100 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:20:05.100 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:05.100 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:20:05.100 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:20:05.101 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:05.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:05.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:05.102 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:20:05.102 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:05.102 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:20:05.102 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:20:05.102 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:05.102 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:20:05.103 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:05.103 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:20:05.103 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:20:05.103 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:05.103 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:05.103 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:20:05.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:05.104 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:20:05.104 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:20:05.104 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:20:05.104 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:05.104 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:20:05.105 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604945908.145521_359.1 target=1_1604945825.625804_254.1 2020-11-09T18:20:10.107 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:20:10.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:20:10.108 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:20:10.149 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:20:10.149 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:20:10.417 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:20:10.418 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:20:10.418 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:20:10.418 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:20:10.418 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:20:10.419 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:20:10.419 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:20:10.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:10.419 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:20:10.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:10.419 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:20:10.419 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:10.420 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:20:10.420 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945908.145521_359.1", 2020-11-09T18:20:10.420 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:10.420 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:20:10.420 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:10.420 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:18:28.145521Z", 2020-11-09T18:20:10.421 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:20:10.421 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:10.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:10.421 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:10.421 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:20:10.421 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:10.421 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:20:10.422 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:20:10.422 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:10.422 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:20:10.422 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:10.422 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:20:10.422 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:20:10.423 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:10.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:10.423 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:10.423 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:20:10.423 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:10.423 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:20:10.423 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:20:10.424 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:10.424 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:20:10.424 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:10.424 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:20:10.424 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:20:10.424 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:10.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:10.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:10.425 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:20:10.425 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:10.425 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:20:10.425 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:20:10.425 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:10.425 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:20:10.426 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:10.426 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:20:10.426 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:20:10.426 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:10.426 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:10.426 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:20:10.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:10.427 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:20:10.427 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:20:10.427 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:20:10.427 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:10.427 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:20:10.428 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-09T18:20:10.429 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-09T18:20:10.429 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:20:10.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:20:10.430 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:20:10.473 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:20:10.473 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:20:10.741 INFO:teuthology.orchestra.run.smithi051.stdout:{ 2020-11-09T18:20:10.742 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_status": { 2020-11-09T18:20:10.742 INFO:teuthology.orchestra.run.smithi051.stdout: "info": { 2020-11-09T18:20:10.743 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "sync", 2020-11-09T18:20:10.743 INFO:teuthology.orchestra.run.smithi051.stdout: "num_shards": 4, 2020-11-09T18:20:10.743 INFO:teuthology.orchestra.run.smithi051.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:20:10.743 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 4 2020-11-09T18:20:10.744 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:20:10.744 INFO:teuthology.orchestra.run.smithi051.stdout: "markers": [ 2020-11-09T18:20:10.744 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:20:10.745 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 0, 2020-11-09T18:20:10.745 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:20:10.745 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:20:10.745 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945908.145521_359.1", 2020-11-09T18:20:10.746 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:20:10.746 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 1, 2020-11-09T18:20:10.746 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:20:10.746 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:18:28.145521Z", 2020-11-09T18:20:10.747 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 4 2020-11-09T18:20:10.747 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:20:10.747 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:20:10.748 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:20:10.748 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 1, 2020-11-09T18:20:10.748 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:20:10.748 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:20:10.749 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:20:10.749 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:20:10.750 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:20:10.750 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:20:10.750 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:20:10.752 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:20:10.752 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:20:10.752 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:20:10.753 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:20:10.754 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 2, 2020-11-09T18:20:10.754 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:20:10.754 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:20:10.755 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:20:10.755 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:20:10.755 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:20:10.755 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:20:10.755 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:20:10.755 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:20:10.756 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:20:10.756 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:20:10.756 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:20:10.756 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 3, 2020-11-09T18:20:10.756 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:20:10.756 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:20:10.756 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:20:10.757 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:20:10.757 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:20:10.757 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:20:10.757 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:20:10.757 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:20:10.757 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:20:10.757 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:20:10.758 INFO:teuthology.orchestra.run.smithi051.stdout: ] 2020-11-09T18:20:10.758 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:20:10.758 INFO:teuthology.orchestra.run.smithi051.stdout: "full_sync": { 2020-11-09T18:20:10.758 INFO:teuthology.orchestra.run.smithi051.stdout: "total": 1, 2020-11-09T18:20:10.758 INFO:teuthology.orchestra.run.smithi051.stdout: "complete": 1 2020-11-09T18:20:10.758 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:20:10.758 INFO:teuthology.orchestra.run.smithi051.stdout:} 2020-11-09T18:20:10.759 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-09T18:20:10.760 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-09T18:20:10.760 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:20:10.761 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:20:10.761 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:20:10.803 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:20:10.803 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:20:11.070 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:20:11.071 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:20:11.071 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:20:11.071 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:20:11.071 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:20:11.071 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:20:11.072 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:20:11.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:11.072 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:20:11.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:11.072 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:20:11.072 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:11.073 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:20:11.073 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945908.145521_359.1", 2020-11-09T18:20:11.073 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:11.073 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:20:11.073 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:11.073 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:18:28.145521Z", 2020-11-09T18:20:11.073 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:20:11.074 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:11.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:11.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:11.074 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:20:11.074 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:11.074 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:20:11.074 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:20:11.075 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:11.075 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:20:11.075 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:11.075 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:20:11.075 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:20:11.075 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:11.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:11.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:11.076 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:20:11.076 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:11.076 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:20:11.076 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:20:11.076 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:11.076 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:20:11.077 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:11.077 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:20:11.077 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:20:11.077 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:11.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:11.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:11.077 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:20:11.078 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:11.078 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:20:11.078 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:20:11.078 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:11.078 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:20:11.078 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:11.078 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:20:11.079 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:20:11.079 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:11.079 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:11.079 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:20:11.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:11.079 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:20:11.079 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:20:11.080 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:20:11.080 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:11.080 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:20:11.081 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-09T18:20:11.081 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'ymzeui-56', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:20:11.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'ymzeui-56', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:20:11.082 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket ymzeui-56 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-09T18:20:11.132 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:20:11.132 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:20:11.539 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'ymzeui-57', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:20:11.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'ymzeui-57', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:20:11.540 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket ymzeui-57 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-09T18:20:11.579 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:20:11.580 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:20:11.938 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'ymzeui-58', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:20:11.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'ymzeui-58', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T18:20:11.939 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket ymzeui-58 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-09T18:20:12.026 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:20:12.026 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:20:12.359 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:20:12.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:20:12.361 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:20:12.439 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:20:12.439 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:20:12.721 INFO:teuthology.orchestra.run.smithi051.stdout:[ 2020-11-09T18:20:12.721 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000078", 2020-11-09T18:20:12.721 INFO:teuthology.orchestra.run.smithi051.stdout: "meta.history", 2020-11-09T18:20:12.722 INFO:teuthology.orchestra.run.smithi051.stdout: "datalog.sync-status.shard.a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6.3", 2020-11-09T18:20:12.722 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000070", 2020-11-09T18:20:12.722 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000104", 2020-11-09T18:20:12.722 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.29-ymzeui-29", 2020-11-09T18:20:12.722 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000026", 2020-11-09T18:20:12.722 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000028", 2020-11-09T18:20:12.723 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000040", 2020-11-09T18:20:12.723 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.28-ymzeui-28", 2020-11-09T18:20:12.723 INFO:teuthology.orchestra.run.smithi051.stdout: "meta.full-sync.index.0", 2020-11-09T18:20:12.723 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.19-ymzeui-19", 2020-11-09T18:20:12.723 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.3-ymzeui-3", 2020-11-09T18:20:12.723 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000015", 2020-11-09T18:20:12.723 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000069", 2020-11-09T18:20:12.724 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000095", 2020-11-09T18:20:12.724 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000003", 2020-11-09T18:20:12.724 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000047", 2020-11-09T18:20:12.724 INFO:teuthology.orchestra.run.smithi051.stdout: "data.full-sync.index.4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc.2", 2020-11-09T18:20:12.724 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000035", 2020-11-09T18:20:12.724 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000037", 2020-11-09T18:20:12.724 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.37-ymzeui-37", 2020-11-09T18:20:12.724 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000024", 2020-11-09T18:20:12.725 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000000", 2020-11-09T18:20:12.725 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000031", 2020-11-09T18:20:12.725 INFO:teuthology.orchestra.run.smithi051.stdout: "data_log.3", 2020-11-09T18:20:12.725 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4748.4-ymzeui-53", 2020-11-09T18:20:12.725 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000076", 2020-11-09T18:20:12.725 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000113", 2020-11-09T18:20:12.725 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000005", 2020-11-09T18:20:12.726 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000011", 2020-11-09T18:20:12.726 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000058", 2020-11-09T18:20:12.726 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000052", 2020-11-09T18:20:12.726 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.41-ymzeui-41", 2020-11-09T18:20:12.726 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4545.4-ymzeui-47", 2020-11-09T18:20:12.726 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000088", 2020-11-09T18:20:12.726 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000080", 2020-11-09T18:20:12.727 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000090", 2020-11-09T18:20:12.727 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.32-ymzeui-32", 2020-11-09T18:20:12.727 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4545.5-ymzeui-48", 2020-11-09T18:20:12.727 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000110", 2020-11-09T18:20:12.727 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000096", 2020-11-09T18:20:12.727 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000087", 2020-11-09T18:20:12.727 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.6-ymzeui-6", 2020-11-09T18:20:12.728 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000008", 2020-11-09T18:20:12.728 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.24-ymzeui-24", 2020-11-09T18:20:12.728 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.10-ymzeui-10", 2020-11-09T18:20:12.728 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000006", 2020-11-09T18:20:12.728 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000029", 2020-11-09T18:20:12.728 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000089", 2020-11-09T18:20:12.728 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000022", 2020-11-09T18:20:12.729 INFO:teuthology.orchestra.run.smithi051.stdout: "bilog.trim", 2020-11-09T18:20:12.729 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000016", 2020-11-09T18:20:12.729 INFO:teuthology.orchestra.run.smithi051.stdout: "data.full-sync.index.a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6.1", 2020-11-09T18:20:12.729 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000041", 2020-11-09T18:20:12.729 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000018", 2020-11-09T18:20:12.729 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000092", 2020-11-09T18:20:12.729 INFO:teuthology.orchestra.run.smithi051.stdout: "datalog.sync-status.shard.4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc.2", 2020-11-09T18:20:12.730 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000014", 2020-11-09T18:20:12.730 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000112", 2020-11-09T18:20:12.730 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000007", 2020-11-09T18:20:12.730 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000021", 2020-11-09T18:20:12.730 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000064", 2020-11-09T18:20:12.730 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000071", 2020-11-09T18:20:12.730 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000074", 2020-11-09T18:20:12.731 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.23-ymzeui-23", 2020-11-09T18:20:12.731 INFO:teuthology.orchestra.run.smithi051.stdout: "data.full-sync.index.a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6.0", 2020-11-09T18:20:12.731 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000081", 2020-11-09T18:20:12.731 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000009", 2020-11-09T18:20:12.731 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4545.2-ymzeui-45", 2020-11-09T18:20:12.731 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000121", 2020-11-09T18:20:12.731 INFO:teuthology.orchestra.run.smithi051.stdout: "data_log.1", 2020-11-09T18:20:12.732 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.34-ymzeui-34", 2020-11-09T18:20:12.732 INFO:teuthology.orchestra.run.smithi051.stdout: "data_log.3.Q5FoqNHzXxn.0", 2020-11-09T18:20:12.732 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000125", 2020-11-09T18:20:12.732 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000082", 2020-11-09T18:20:12.732 INFO:teuthology.orchestra.run.smithi051.stdout: "meta.full-sync.index.1", 2020-11-09T18:20:12.732 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000105", 2020-11-09T18:20:12.732 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000059", 2020-11-09T18:20:12.733 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000077", 2020-11-09T18:20:12.733 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000032", 2020-11-09T18:20:12.733 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000053", 2020-11-09T18:20:12.733 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.39-ymzeui-39", 2020-11-09T18:20:12.733 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000091", 2020-11-09T18:20:12.733 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000065", 2020-11-09T18:20:12.733 INFO:teuthology.orchestra.run.smithi051.stdout: "datalog.sync-status.4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc", 2020-11-09T18:20:12.733 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000083", 2020-11-09T18:20:12.734 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000010", 2020-11-09T18:20:12.734 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4748.6-ymzeui-55", 2020-11-09T18:20:12.734 INFO:teuthology.orchestra.run.smithi051.stdout: "datalog.sync-status.shard.4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc.1", 2020-11-09T18:20:12.734 INFO:teuthology.orchestra.run.smithi051.stdout: "mdlog.sync-status.shard.2", 2020-11-09T18:20:12.734 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.25-ymzeui-25", 2020-11-09T18:20:12.734 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000045", 2020-11-09T18:20:12.734 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000002", 2020-11-09T18:20:12.735 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000116", 2020-11-09T18:20:12.735 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000034", 2020-11-09T18:20:12.735 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000101", 2020-11-09T18:20:12.735 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.30-ymzeui-30", 2020-11-09T18:20:12.735 INFO:teuthology.orchestra.run.smithi051.stdout: "mdlog.sync-status.shard.1", 2020-11-09T18:20:12.735 INFO:teuthology.orchestra.run.smithi051.stdout: "datalog.sync-status.shard.4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc.0", 2020-11-09T18:20:12.735 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.5-ymzeui-5", 2020-11-09T18:20:12.736 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000079", 2020-11-09T18:20:12.736 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000049", 2020-11-09T18:20:12.736 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000117", 2020-11-09T18:20:12.736 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000044", 2020-11-09T18:20:12.736 INFO:teuthology.orchestra.run.smithi051.stdout: "datalog.sync-status.shard.a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6.0", 2020-11-09T18:20:12.736 INFO:teuthology.orchestra.run.smithi051.stdout: "datalog.sync-status.a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6", 2020-11-09T18:20:12.737 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000066", 2020-11-09T18:20:12.737 INFO:teuthology.orchestra.run.smithi051.stdout: "data_log.2.TQJ7p+7lAMX.0", 2020-11-09T18:20:12.737 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.17-ymzeui-17", 2020-11-09T18:20:12.737 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.2-ymzeui-2", 2020-11-09T18:20:12.737 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000068", 2020-11-09T18:20:12.737 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000085", 2020-11-09T18:20:12.737 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc.4649.2-ymzeui-43", 2020-11-09T18:20:12.738 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.35-ymzeui-35", 2020-11-09T18:20:12.738 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000073", 2020-11-09T18:20:12.738 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000038", 2020-11-09T18:20:12.738 INFO:teuthology.orchestra.run.smithi051.stdout: "meta.full-sync.index.3", 2020-11-09T18:20:12.738 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000118", 2020-11-09T18:20:12.738 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.14-ymzeui-14", 2020-11-09T18:20:12.738 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4748.9-ymzeui-58", 2020-11-09T18:20:12.739 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000036", 2020-11-09T18:20:12.739 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000103", 2020-11-09T18:20:12.739 INFO:teuthology.orchestra.run.smithi051.stdout: "mdlog.sync-status.shard.3", 2020-11-09T18:20:12.739 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000119", 2020-11-09T18:20:12.739 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000098", 2020-11-09T18:20:12.739 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.31-ymzeui-31", 2020-11-09T18:20:12.739 INFO:teuthology.orchestra.run.smithi051.stdout: "data_log.0.wpe4HJuNyfH.0", 2020-11-09T18:20:12.740 INFO:teuthology.orchestra.run.smithi051.stdout: "mdlog.sync-status.shard.0", 2020-11-09T18:20:12.740 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000027", 2020-11-09T18:20:12.740 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4748.1-ymzeui-50", 2020-11-09T18:20:12.740 INFO:teuthology.orchestra.run.smithi051.stdout: "mdlog.sync-status", 2020-11-09T18:20:12.740 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000019", 2020-11-09T18:20:12.740 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc.4649.1-ymzeui-42", 2020-11-09T18:20:12.740 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000039", 2020-11-09T18:20:12.741 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.26-ymzeui-26", 2020-11-09T18:20:12.741 INFO:teuthology.orchestra.run.smithi051.stdout: "data.full-sync.index.4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc.3", 2020-11-09T18:20:12.741 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000100", 2020-11-09T18:20:12.741 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.4-ymzeui-4", 2020-11-09T18:20:12.741 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000093", 2020-11-09T18:20:12.741 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000004", 2020-11-09T18:20:12.741 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000063", 2020-11-09T18:20:12.742 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000122", 2020-11-09T18:20:12.742 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.13-ymzeui-13", 2020-11-09T18:20:12.742 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000057", 2020-11-09T18:20:12.742 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4748.8-ymzeui-57", 2020-11-09T18:20:12.742 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000054", 2020-11-09T18:20:12.742 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000114", 2020-11-09T18:20:12.742 INFO:teuthology.orchestra.run.smithi051.stdout: "data_log.0", 2020-11-09T18:20:12.743 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4748.7-ymzeui-56", 2020-11-09T18:20:12.743 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000012", 2020-11-09T18:20:12.743 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.33-ymzeui-33", 2020-11-09T18:20:12.743 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.40-ymzeui-40", 2020-11-09T18:20:12.743 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000084", 2020-11-09T18:20:12.743 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000043", 2020-11-09T18:20:12.743 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000111", 2020-11-09T18:20:12.743 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000048", 2020-11-09T18:20:12.744 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000020", 2020-11-09T18:20:12.744 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.9-ymzeui-9", 2020-11-09T18:20:12.744 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000099", 2020-11-09T18:20:12.744 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000056", 2020-11-09T18:20:12.744 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.21-ymzeui-21", 2020-11-09T18:20:12.744 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000072", 2020-11-09T18:20:12.744 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000062", 2020-11-09T18:20:12.745 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000115", 2020-11-09T18:20:12.745 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000033", 2020-11-09T18:20:12.745 INFO:teuthology.orchestra.run.smithi051.stdout: "data.full-sync.index.4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc.1", 2020-11-09T18:20:12.745 INFO:teuthology.orchestra.run.smithi051.stdout: "data.full-sync.index.a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6.3", 2020-11-09T18:20:12.745 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000025", 2020-11-09T18:20:12.745 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000123", 2020-11-09T18:20:12.745 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.20-ymzeui-20", 2020-11-09T18:20:12.745 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000108", 2020-11-09T18:20:12.746 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.27-ymzeui-27", 2020-11-09T18:20:12.746 INFO:teuthology.orchestra.run.smithi051.stdout: "meta.log.f6258aba-300a-4d5d-ad93-a7c198a0e0a1.0", 2020-11-09T18:20:12.746 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000094", 2020-11-09T18:20:12.746 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000060", 2020-11-09T18:20:12.746 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.18-ymzeui-18", 2020-11-09T18:20:12.746 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000109", 2020-11-09T18:20:12.747 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.11-ymzeui-11", 2020-11-09T18:20:12.747 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000013", 2020-11-09T18:20:12.747 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.1-ymzeui-1", 2020-11-09T18:20:12.747 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000051", 2020-11-09T18:20:12.747 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000106", 2020-11-09T18:20:12.747 INFO:teuthology.orchestra.run.smithi051.stdout: "data.full-sync.index.a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6.2", 2020-11-09T18:20:12.747 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000017", 2020-11-09T18:20:12.747 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.16-ymzeui-16", 2020-11-09T18:20:12.748 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000023", 2020-11-09T18:20:12.748 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4748.3-ymzeui-52", 2020-11-09T18:20:12.748 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.22-ymzeui-22", 2020-11-09T18:20:12.748 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000107", 2020-11-09T18:20:12.748 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000046", 2020-11-09T18:20:12.748 INFO:teuthology.orchestra.run.smithi051.stdout: "meta.full-sync.index.2", 2020-11-09T18:20:12.748 INFO:teuthology.orchestra.run.smithi051.stdout: "data_log.2", 2020-11-09T18:20:12.749 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000067", 2020-11-09T18:20:12.749 INFO:teuthology.orchestra.run.smithi051.stdout: "datalog.sync-status.shard.a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6.1", 2020-11-09T18:20:12.749 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000086", 2020-11-09T18:20:12.749 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4748.2-ymzeui-51", 2020-11-09T18:20:12.749 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.7-ymzeui-7", 2020-11-09T18:20:12.749 INFO:teuthology.orchestra.run.smithi051.stdout: "datalog.sync-status.shard.4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc.3", 2020-11-09T18:20:12.749 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.38-ymzeui-38", 2020-11-09T18:20:12.750 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.15-ymzeui-15", 2020-11-09T18:20:12.750 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4545.1-ymzeui-44", 2020-11-09T18:20:12.750 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4655.1-ymzeui-49", 2020-11-09T18:20:12.750 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4545.3-ymzeui-46", 2020-11-09T18:20:12.750 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.8-ymzeui-8", 2020-11-09T18:20:12.750 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000050", 2020-11-09T18:20:12.750 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000102", 2020-11-09T18:20:12.751 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4748.5-ymzeui-54", 2020-11-09T18:20:12.751 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.36-ymzeui-36", 2020-11-09T18:20:12.751 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000124", 2020-11-09T18:20:12.751 INFO:teuthology.orchestra.run.smithi051.stdout: "datalog.sync-status.shard.a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6.2", 2020-11-09T18:20:12.751 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000001", 2020-11-09T18:20:12.751 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000126", 2020-11-09T18:20:12.751 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000055", 2020-11-09T18:20:12.751 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000061", 2020-11-09T18:20:12.752 INFO:teuthology.orchestra.run.smithi051.stdout: "data_log.1.Q2uEj45zR2p.0", 2020-11-09T18:20:12.752 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000097", 2020-11-09T18:20:12.752 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.12-ymzeui-12", 2020-11-09T18:20:12.752 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000042", 2020-11-09T18:20:12.752 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000075", 2020-11-09T18:20:12.752 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000120", 2020-11-09T18:20:12.752 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000030", 2020-11-09T18:20:12.753 INFO:teuthology.orchestra.run.smithi051.stdout: "data.full-sync.index.4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc.0" 2020-11-09T18:20:12.753 INFO:teuthology.orchestra.run.smithi051.stdout:] 2020-11-09T18:20:12.753 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:20:12.755 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:20:12.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:20:12.756 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:20:12.792 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:20:12.793 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:20:13.064 INFO:teuthology.orchestra.run.smithi078.stdout:[ 2020-11-09T18:20:13.065 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000078", 2020-11-09T18:20:13.065 INFO:teuthology.orchestra.run.smithi078.stdout: "meta.history", 2020-11-09T18:20:13.065 INFO:teuthology.orchestra.run.smithi078.stdout: "datalog.sync-status.shard.a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6.3", 2020-11-09T18:20:13.065 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000070", 2020-11-09T18:20:13.065 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000104", 2020-11-09T18:20:13.066 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000026", 2020-11-09T18:20:13.066 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000028", 2020-11-09T18:20:13.066 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000040", 2020-11-09T18:20:13.066 INFO:teuthology.orchestra.run.smithi078.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.28-ymzeui-28", 2020-11-09T18:20:13.066 INFO:teuthology.orchestra.run.smithi078.stdout: "meta.full-sync.index.0", 2020-11-09T18:20:13.066 INFO:teuthology.orchestra.run.smithi078.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.3-ymzeui-3", 2020-11-09T18:20:13.067 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000015", 2020-11-09T18:20:13.067 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000069", 2020-11-09T18:20:13.067 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000095", 2020-11-09T18:20:13.067 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000003", 2020-11-09T18:20:13.067 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000047", 2020-11-09T18:20:13.067 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000035", 2020-11-09T18:20:13.067 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000037", 2020-11-09T18:20:13.068 INFO:teuthology.orchestra.run.smithi078.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.37-ymzeui-37", 2020-11-09T18:20:13.068 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000024", 2020-11-09T18:20:13.068 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000000", 2020-11-09T18:20:13.068 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000031", 2020-11-09T18:20:13.068 INFO:teuthology.orchestra.run.smithi078.stdout: "data_log.3", 2020-11-09T18:20:13.068 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000076", 2020-11-09T18:20:13.068 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000113", 2020-11-09T18:20:13.069 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000005", 2020-11-09T18:20:13.069 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000011", 2020-11-09T18:20:13.069 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000058", 2020-11-09T18:20:13.069 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000052", 2020-11-09T18:20:13.069 INFO:teuthology.orchestra.run.smithi078.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.41-ymzeui-41", 2020-11-09T18:20:13.069 INFO:teuthology.orchestra.run.smithi078.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4545.4-ymzeui-47", 2020-11-09T18:20:13.070 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000088", 2020-11-09T18:20:13.070 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000080", 2020-11-09T18:20:13.070 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000090", 2020-11-09T18:20:13.070 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000110", 2020-11-09T18:20:13.070 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000096", 2020-11-09T18:20:13.070 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000087", 2020-11-09T18:20:13.070 INFO:teuthology.orchestra.run.smithi078.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.6-ymzeui-6", 2020-11-09T18:20:13.071 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000008", 2020-11-09T18:20:13.071 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000006", 2020-11-09T18:20:13.071 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000029", 2020-11-09T18:20:13.071 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000089", 2020-11-09T18:20:13.071 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000022", 2020-11-09T18:20:13.071 INFO:teuthology.orchestra.run.smithi078.stdout: "bilog.trim", 2020-11-09T18:20:13.071 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000016", 2020-11-09T18:20:13.072 INFO:teuthology.orchestra.run.smithi078.stdout: "data.full-sync.index.a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6.1", 2020-11-09T18:20:13.072 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000041", 2020-11-09T18:20:13.072 INFO:teuthology.orchestra.run.smithi078.stdout: "data.full-sync.index.11f516a7-3288-49a6-9a89-b45ac1dae91a.3", 2020-11-09T18:20:13.072 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000018", 2020-11-09T18:20:13.072 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000092", 2020-11-09T18:20:13.072 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000014", 2020-11-09T18:20:13.072 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000112", 2020-11-09T18:20:13.072 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000007", 2020-11-09T18:20:13.073 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000021", 2020-11-09T18:20:13.073 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000064", 2020-11-09T18:20:13.073 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000071", 2020-11-09T18:20:13.073 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000074", 2020-11-09T18:20:13.073 INFO:teuthology.orchestra.run.smithi078.stdout: "data.full-sync.index.a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6.0", 2020-11-09T18:20:13.073 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000081", 2020-11-09T18:20:13.073 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000009", 2020-11-09T18:20:13.074 INFO:teuthology.orchestra.run.smithi078.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4545.2-ymzeui-45", 2020-11-09T18:20:13.074 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000121", 2020-11-09T18:20:13.074 INFO:teuthology.orchestra.run.smithi078.stdout: "data_log.1", 2020-11-09T18:20:13.074 INFO:teuthology.orchestra.run.smithi078.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.34-ymzeui-34", 2020-11-09T18:20:13.074 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000125", 2020-11-09T18:20:13.074 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000082", 2020-11-09T18:20:13.074 INFO:teuthology.orchestra.run.smithi078.stdout: "meta.full-sync.index.1", 2020-11-09T18:20:13.075 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000105", 2020-11-09T18:20:13.075 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000059", 2020-11-09T18:20:13.075 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000077", 2020-11-09T18:20:13.075 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000032", 2020-11-09T18:20:13.075 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000053", 2020-11-09T18:20:13.075 INFO:teuthology.orchestra.run.smithi078.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.39-ymzeui-39", 2020-11-09T18:20:13.075 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000091", 2020-11-09T18:20:13.076 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000065", 2020-11-09T18:20:13.076 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000083", 2020-11-09T18:20:13.076 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000010", 2020-11-09T18:20:13.076 INFO:teuthology.orchestra.run.smithi078.stdout: "mdlog.sync-status.shard.2", 2020-11-09T18:20:13.076 INFO:teuthology.orchestra.run.smithi078.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.25-ymzeui-25", 2020-11-09T18:20:13.076 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000045", 2020-11-09T18:20:13.076 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000002", 2020-11-09T18:20:13.077 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000116", 2020-11-09T18:20:13.077 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000034", 2020-11-09T18:20:13.077 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000101", 2020-11-09T18:20:13.077 INFO:teuthology.orchestra.run.smithi078.stdout: "mdlog.sync-status.shard.1", 2020-11-09T18:20:13.077 INFO:teuthology.orchestra.run.smithi078.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.5-ymzeui-5", 2020-11-09T18:20:13.077 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000079", 2020-11-09T18:20:13.077 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000049", 2020-11-09T18:20:13.077 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000117", 2020-11-09T18:20:13.078 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000044", 2020-11-09T18:20:13.078 INFO:teuthology.orchestra.run.smithi078.stdout: "datalog.sync-status.shard.a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6.0", 2020-11-09T18:20:13.078 INFO:teuthology.orchestra.run.smithi078.stdout: "datalog.sync-status.a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6", 2020-11-09T18:20:13.078 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000066", 2020-11-09T18:20:13.078 INFO:teuthology.orchestra.run.smithi078.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.17-ymzeui-17", 2020-11-09T18:20:13.078 INFO:teuthology.orchestra.run.smithi078.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.2-ymzeui-2", 2020-11-09T18:20:13.078 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000068", 2020-11-09T18:20:13.079 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000085", 2020-11-09T18:20:13.079 INFO:teuthology.orchestra.run.smithi078.stdout: "2020-11-09-18-4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc.4649.2-ymzeui-43", 2020-11-09T18:20:13.079 INFO:teuthology.orchestra.run.smithi078.stdout: "data.full-sync.index.11f516a7-3288-49a6-9a89-b45ac1dae91a.0", 2020-11-09T18:20:13.079 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000073", 2020-11-09T18:20:13.079 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000038", 2020-11-09T18:20:13.079 INFO:teuthology.orchestra.run.smithi078.stdout: "meta.full-sync.index.3", 2020-11-09T18:20:13.079 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000118", 2020-11-09T18:20:13.080 INFO:teuthology.orchestra.run.smithi078.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.14-ymzeui-14", 2020-11-09T18:20:13.080 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000036", 2020-11-09T18:20:13.080 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000103", 2020-11-09T18:20:13.080 INFO:teuthology.orchestra.run.smithi078.stdout: "mdlog.sync-status.shard.3", 2020-11-09T18:20:13.080 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000119", 2020-11-09T18:20:13.080 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000098", 2020-11-09T18:20:13.080 INFO:teuthology.orchestra.run.smithi078.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.31-ymzeui-31", 2020-11-09T18:20:13.080 INFO:teuthology.orchestra.run.smithi078.stdout: "mdlog.sync-status.shard.0", 2020-11-09T18:20:13.081 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000027", 2020-11-09T18:20:13.081 INFO:teuthology.orchestra.run.smithi078.stdout: "mdlog.sync-status", 2020-11-09T18:20:13.081 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000019", 2020-11-09T18:20:13.081 INFO:teuthology.orchestra.run.smithi078.stdout: "2020-11-09-18-4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc.4649.1-ymzeui-42", 2020-11-09T18:20:13.081 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000039", 2020-11-09T18:20:13.081 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000100", 2020-11-09T18:20:13.081 INFO:teuthology.orchestra.run.smithi078.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.4-ymzeui-4", 2020-11-09T18:20:13.082 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000093", 2020-11-09T18:20:13.082 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000004", 2020-11-09T18:20:13.082 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000063", 2020-11-09T18:20:13.082 INFO:teuthology.orchestra.run.smithi078.stdout: "datalog.sync-status.shard.11f516a7-3288-49a6-9a89-b45ac1dae91a.3", 2020-11-09T18:20:13.082 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000122", 2020-11-09T18:20:13.082 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000057", 2020-11-09T18:20:13.082 INFO:teuthology.orchestra.run.smithi078.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4748.8-ymzeui-57", 2020-11-09T18:20:13.083 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000054", 2020-11-09T18:20:13.083 INFO:teuthology.orchestra.run.smithi078.stdout: "data.full-sync.index.11f516a7-3288-49a6-9a89-b45ac1dae91a.2", 2020-11-09T18:20:13.083 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000114", 2020-11-09T18:20:13.083 INFO:teuthology.orchestra.run.smithi078.stdout: "data_log.0", 2020-11-09T18:20:13.083 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000012", 2020-11-09T18:20:13.083 INFO:teuthology.orchestra.run.smithi078.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.40-ymzeui-40", 2020-11-09T18:20:13.083 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000084", 2020-11-09T18:20:13.084 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000043", 2020-11-09T18:20:13.084 INFO:teuthology.orchestra.run.smithi078.stdout: "data.full-sync.index.11f516a7-3288-49a6-9a89-b45ac1dae91a.1", 2020-11-09T18:20:13.084 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000111", 2020-11-09T18:20:13.084 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000048", 2020-11-09T18:20:13.084 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000020", 2020-11-09T18:20:13.084 INFO:teuthology.orchestra.run.smithi078.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.9-ymzeui-9", 2020-11-09T18:20:13.084 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000099", 2020-11-09T18:20:13.084 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000056", 2020-11-09T18:20:13.085 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000072", 2020-11-09T18:20:13.085 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000062", 2020-11-09T18:20:13.085 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000115", 2020-11-09T18:20:13.085 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000033", 2020-11-09T18:20:13.085 INFO:teuthology.orchestra.run.smithi078.stdout: "data.full-sync.index.a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6.3", 2020-11-09T18:20:13.085 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000025", 2020-11-09T18:20:13.085 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000123", 2020-11-09T18:20:13.086 INFO:teuthology.orchestra.run.smithi078.stdout: "datalog.sync-status.shard.11f516a7-3288-49a6-9a89-b45ac1dae91a.0", 2020-11-09T18:20:13.086 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000108", 2020-11-09T18:20:13.086 INFO:teuthology.orchestra.run.smithi078.stdout: "meta.log.f6258aba-300a-4d5d-ad93-a7c198a0e0a1.0", 2020-11-09T18:20:13.086 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000094", 2020-11-09T18:20:13.086 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000060", 2020-11-09T18:20:13.086 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000109", 2020-11-09T18:20:13.086 INFO:teuthology.orchestra.run.smithi078.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.11-ymzeui-11", 2020-11-09T18:20:13.087 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000013", 2020-11-09T18:20:13.087 INFO:teuthology.orchestra.run.smithi078.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.1-ymzeui-1", 2020-11-09T18:20:13.087 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000051", 2020-11-09T18:20:13.087 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000106", 2020-11-09T18:20:13.087 INFO:teuthology.orchestra.run.smithi078.stdout: "data.full-sync.index.a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6.2", 2020-11-09T18:20:13.087 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000017", 2020-11-09T18:20:13.087 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000023", 2020-11-09T18:20:13.087 INFO:teuthology.orchestra.run.smithi078.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.22-ymzeui-22", 2020-11-09T18:20:13.088 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000107", 2020-11-09T18:20:13.088 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000046", 2020-11-09T18:20:13.088 INFO:teuthology.orchestra.run.smithi078.stdout: "meta.full-sync.index.2", 2020-11-09T18:20:13.088 INFO:teuthology.orchestra.run.smithi078.stdout: "data_log.2", 2020-11-09T18:20:13.088 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000067", 2020-11-09T18:20:13.088 INFO:teuthology.orchestra.run.smithi078.stdout: "datalog.sync-status.shard.a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6.1", 2020-11-09T18:20:13.088 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000086", 2020-11-09T18:20:13.089 INFO:teuthology.orchestra.run.smithi078.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4748.2-ymzeui-51", 2020-11-09T18:20:13.089 INFO:teuthology.orchestra.run.smithi078.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.7-ymzeui-7", 2020-11-09T18:20:13.089 INFO:teuthology.orchestra.run.smithi078.stdout: "datalog.sync-status.11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:20:13.089 INFO:teuthology.orchestra.run.smithi078.stdout: "datalog.sync-status.shard.11f516a7-3288-49a6-9a89-b45ac1dae91a.1", 2020-11-09T18:20:13.089 INFO:teuthology.orchestra.run.smithi078.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4545.1-ymzeui-44", 2020-11-09T18:20:13.089 INFO:teuthology.orchestra.run.smithi078.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.8-ymzeui-8", 2020-11-09T18:20:13.089 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000050", 2020-11-09T18:20:13.090 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000102", 2020-11-09T18:20:13.090 INFO:teuthology.orchestra.run.smithi078.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4748.5-ymzeui-54", 2020-11-09T18:20:13.090 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000124", 2020-11-09T18:20:13.090 INFO:teuthology.orchestra.run.smithi078.stdout: "datalog.sync-status.shard.a1fd2df3-09c0-4b1c-81a7-16b6b2b5cbd6.2", 2020-11-09T18:20:13.090 INFO:teuthology.orchestra.run.smithi078.stdout: "datalog.sync-status.shard.11f516a7-3288-49a6-9a89-b45ac1dae91a.2", 2020-11-09T18:20:13.090 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000001", 2020-11-09T18:20:13.090 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000126", 2020-11-09T18:20:13.091 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000055", 2020-11-09T18:20:13.091 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000061", 2020-11-09T18:20:13.091 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000097", 2020-11-09T18:20:13.091 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000042", 2020-11-09T18:20:13.091 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000075", 2020-11-09T18:20:13.091 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000120", 2020-11-09T18:20:13.091 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000030" 2020-11-09T18:20:13.091 INFO:teuthology.orchestra.run.smithi078.stdout:] 2020-11-09T18:20:13.092 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:20:13.095 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:20:13.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:20:13.096 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:20:13.132 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:20:13.133 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:20:13.418 INFO:teuthology.orchestra.run.smithi051.stdout:[ 2020-11-09T18:20:13.418 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000078", 2020-11-09T18:20:13.418 INFO:teuthology.orchestra.run.smithi051.stdout: "meta.history", 2020-11-09T18:20:13.418 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000070", 2020-11-09T18:20:13.419 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000104", 2020-11-09T18:20:13.419 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.29-ymzeui-29", 2020-11-09T18:20:13.419 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000026", 2020-11-09T18:20:13.419 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000028", 2020-11-09T18:20:13.419 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000040", 2020-11-09T18:20:13.419 INFO:teuthology.orchestra.run.smithi051.stdout: "meta.full-sync.index.0", 2020-11-09T18:20:13.419 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.3-ymzeui-3", 2020-11-09T18:20:13.420 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000015", 2020-11-09T18:20:13.420 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000069", 2020-11-09T18:20:13.420 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000095", 2020-11-09T18:20:13.420 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000003", 2020-11-09T18:20:13.420 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000047", 2020-11-09T18:20:13.420 INFO:teuthology.orchestra.run.smithi051.stdout: "data.full-sync.index.4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc.2", 2020-11-09T18:20:13.421 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000035", 2020-11-09T18:20:13.421 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000037", 2020-11-09T18:20:13.421 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000024", 2020-11-09T18:20:13.421 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000000", 2020-11-09T18:20:13.421 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000031", 2020-11-09T18:20:13.421 INFO:teuthology.orchestra.run.smithi051.stdout: "data_log.3", 2020-11-09T18:20:13.422 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000076", 2020-11-09T18:20:13.422 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000113", 2020-11-09T18:20:13.422 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000005", 2020-11-09T18:20:13.422 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000011", 2020-11-09T18:20:13.422 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000058", 2020-11-09T18:20:13.422 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000052", 2020-11-09T18:20:13.423 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.41-ymzeui-41", 2020-11-09T18:20:13.423 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000088", 2020-11-09T18:20:13.423 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000080", 2020-11-09T18:20:13.423 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000090", 2020-11-09T18:20:13.423 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.32-ymzeui-32", 2020-11-09T18:20:13.423 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4545.5-ymzeui-48", 2020-11-09T18:20:13.423 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000110", 2020-11-09T18:20:13.424 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000096", 2020-11-09T18:20:13.424 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000087", 2020-11-09T18:20:13.424 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.6-ymzeui-6", 2020-11-09T18:20:13.424 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000008", 2020-11-09T18:20:13.424 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000006", 2020-11-09T18:20:13.424 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000029", 2020-11-09T18:20:13.425 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000089", 2020-11-09T18:20:13.425 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000022", 2020-11-09T18:20:13.425 INFO:teuthology.orchestra.run.smithi051.stdout: "bilog.trim", 2020-11-09T18:20:13.425 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000016", 2020-11-09T18:20:13.425 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000041", 2020-11-09T18:20:13.425 INFO:teuthology.orchestra.run.smithi051.stdout: "data.full-sync.index.11f516a7-3288-49a6-9a89-b45ac1dae91a.3", 2020-11-09T18:20:13.425 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000018", 2020-11-09T18:20:13.426 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000092", 2020-11-09T18:20:13.426 INFO:teuthology.orchestra.run.smithi051.stdout: "datalog.sync-status.shard.4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc.2", 2020-11-09T18:20:13.426 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000014", 2020-11-09T18:20:13.426 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000112", 2020-11-09T18:20:13.426 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000007", 2020-11-09T18:20:13.426 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000021", 2020-11-09T18:20:13.426 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000064", 2020-11-09T18:20:13.427 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000071", 2020-11-09T18:20:13.427 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000074", 2020-11-09T18:20:13.427 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.23-ymzeui-23", 2020-11-09T18:20:13.427 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000081", 2020-11-09T18:20:13.427 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000009", 2020-11-09T18:20:13.427 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4545.2-ymzeui-45", 2020-11-09T18:20:13.428 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000121", 2020-11-09T18:20:13.428 INFO:teuthology.orchestra.run.smithi051.stdout: "data_log.1", 2020-11-09T18:20:13.428 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000125", 2020-11-09T18:20:13.428 INFO:teuthology.orchestra.run.smithi051.stdout: "meta.log.bbf9f070-21b9-4940-95e9-d031ba1c647f.0", 2020-11-09T18:20:13.428 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000082", 2020-11-09T18:20:13.428 INFO:teuthology.orchestra.run.smithi051.stdout: "meta.full-sync.index.1", 2020-11-09T18:20:13.428 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000105", 2020-11-09T18:20:13.429 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000059", 2020-11-09T18:20:13.429 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000077", 2020-11-09T18:20:13.429 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000032", 2020-11-09T18:20:13.429 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000053", 2020-11-09T18:20:13.429 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.39-ymzeui-39", 2020-11-09T18:20:13.429 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000091", 2020-11-09T18:20:13.429 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000065", 2020-11-09T18:20:13.430 INFO:teuthology.orchestra.run.smithi051.stdout: "datalog.sync-status.4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc", 2020-11-09T18:20:13.430 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000083", 2020-11-09T18:20:13.430 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000010", 2020-11-09T18:20:13.430 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4748.6-ymzeui-55", 2020-11-09T18:20:13.430 INFO:teuthology.orchestra.run.smithi051.stdout: "datalog.sync-status.shard.4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc.1", 2020-11-09T18:20:13.430 INFO:teuthology.orchestra.run.smithi051.stdout: "mdlog.sync-status.shard.2", 2020-11-09T18:20:13.430 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000045", 2020-11-09T18:20:13.431 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000002", 2020-11-09T18:20:13.431 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000116", 2020-11-09T18:20:13.431 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000034", 2020-11-09T18:20:13.431 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000101", 2020-11-09T18:20:13.431 INFO:teuthology.orchestra.run.smithi051.stdout: "mdlog.sync-status.shard.1", 2020-11-09T18:20:13.431 INFO:teuthology.orchestra.run.smithi051.stdout: "datalog.sync-status.shard.4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc.0", 2020-11-09T18:20:13.432 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.5-ymzeui-5", 2020-11-09T18:20:13.432 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000079", 2020-11-09T18:20:13.432 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000049", 2020-11-09T18:20:13.432 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000117", 2020-11-09T18:20:13.432 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000044", 2020-11-09T18:20:13.432 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000066", 2020-11-09T18:20:13.433 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.2-ymzeui-2", 2020-11-09T18:20:13.433 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000068", 2020-11-09T18:20:13.433 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000085", 2020-11-09T18:20:13.433 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc.4649.2-ymzeui-43", 2020-11-09T18:20:13.433 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.35-ymzeui-35", 2020-11-09T18:20:13.433 INFO:teuthology.orchestra.run.smithi051.stdout: "data.full-sync.index.11f516a7-3288-49a6-9a89-b45ac1dae91a.0", 2020-11-09T18:20:13.434 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000073", 2020-11-09T18:20:13.434 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000038", 2020-11-09T18:20:13.434 INFO:teuthology.orchestra.run.smithi051.stdout: "meta.full-sync.index.3", 2020-11-09T18:20:13.434 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000118", 2020-11-09T18:20:13.434 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4748.9-ymzeui-58", 2020-11-09T18:20:13.434 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000036", 2020-11-09T18:20:13.435 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000103", 2020-11-09T18:20:13.435 INFO:teuthology.orchestra.run.smithi051.stdout: "mdlog.sync-status.shard.3", 2020-11-09T18:20:13.435 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000119", 2020-11-09T18:20:13.435 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000098", 2020-11-09T18:20:13.435 INFO:teuthology.orchestra.run.smithi051.stdout: "mdlog.sync-status.shard.0", 2020-11-09T18:20:13.435 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000027", 2020-11-09T18:20:13.435 INFO:teuthology.orchestra.run.smithi051.stdout: "mdlog.sync-status", 2020-11-09T18:20:13.436 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000019", 2020-11-09T18:20:13.436 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc.4649.1-ymzeui-42", 2020-11-09T18:20:13.436 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000039", 2020-11-09T18:20:13.436 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.26-ymzeui-26", 2020-11-09T18:20:13.436 INFO:teuthology.orchestra.run.smithi051.stdout: "data.full-sync.index.4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc.3", 2020-11-09T18:20:13.436 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000100", 2020-11-09T18:20:13.436 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.4-ymzeui-4", 2020-11-09T18:20:13.437 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000093", 2020-11-09T18:20:13.437 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000004", 2020-11-09T18:20:13.437 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000063", 2020-11-09T18:20:13.437 INFO:teuthology.orchestra.run.smithi051.stdout: "datalog.sync-status.shard.11f516a7-3288-49a6-9a89-b45ac1dae91a.3", 2020-11-09T18:20:13.437 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000122", 2020-11-09T18:20:13.437 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000057", 2020-11-09T18:20:13.437 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000054", 2020-11-09T18:20:13.438 INFO:teuthology.orchestra.run.smithi051.stdout: "data.full-sync.index.11f516a7-3288-49a6-9a89-b45ac1dae91a.2", 2020-11-09T18:20:13.438 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000114", 2020-11-09T18:20:13.438 INFO:teuthology.orchestra.run.smithi051.stdout: "data_log.0", 2020-11-09T18:20:13.438 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000012", 2020-11-09T18:20:13.438 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.40-ymzeui-40", 2020-11-09T18:20:13.438 INFO:teuthology.orchestra.run.smithi051.stdout: "meta.log.a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9.0", 2020-11-09T18:20:13.439 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000084", 2020-11-09T18:20:13.439 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000043", 2020-11-09T18:20:13.439 INFO:teuthology.orchestra.run.smithi051.stdout: "data.full-sync.index.11f516a7-3288-49a6-9a89-b45ac1dae91a.1", 2020-11-09T18:20:13.439 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000111", 2020-11-09T18:20:13.439 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000048", 2020-11-09T18:20:13.439 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000020", 2020-11-09T18:20:13.440 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.9-ymzeui-9", 2020-11-09T18:20:13.440 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000099", 2020-11-09T18:20:13.440 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000056", 2020-11-09T18:20:13.440 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000072", 2020-11-09T18:20:13.440 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000062", 2020-11-09T18:20:13.440 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000115", 2020-11-09T18:20:13.440 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000033", 2020-11-09T18:20:13.441 INFO:teuthology.orchestra.run.smithi051.stdout: "data.full-sync.index.4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc.1", 2020-11-09T18:20:13.441 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000025", 2020-11-09T18:20:13.441 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000123", 2020-11-09T18:20:13.441 INFO:teuthology.orchestra.run.smithi051.stdout: "datalog.sync-status.shard.11f516a7-3288-49a6-9a89-b45ac1dae91a.0", 2020-11-09T18:20:13.441 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000108", 2020-11-09T18:20:13.441 INFO:teuthology.orchestra.run.smithi051.stdout: "meta.log.f6258aba-300a-4d5d-ad93-a7c198a0e0a1.0", 2020-11-09T18:20:13.441 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000094", 2020-11-09T18:20:13.442 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000060", 2020-11-09T18:20:13.442 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.18-ymzeui-18", 2020-11-09T18:20:13.442 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000109", 2020-11-09T18:20:13.442 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000013", 2020-11-09T18:20:13.442 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.1-ymzeui-1", 2020-11-09T18:20:13.442 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000051", 2020-11-09T18:20:13.443 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000106", 2020-11-09T18:20:13.443 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000017", 2020-11-09T18:20:13.443 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000023", 2020-11-09T18:20:13.443 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4748.3-ymzeui-52", 2020-11-09T18:20:13.443 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000107", 2020-11-09T18:20:13.443 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000046", 2020-11-09T18:20:13.444 INFO:teuthology.orchestra.run.smithi051.stdout: "meta.full-sync.index.2", 2020-11-09T18:20:13.444 INFO:teuthology.orchestra.run.smithi051.stdout: "data_log.2", 2020-11-09T18:20:13.444 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000067", 2020-11-09T18:20:13.444 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000086", 2020-11-09T18:20:13.444 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.7-ymzeui-7", 2020-11-09T18:20:13.444 INFO:teuthology.orchestra.run.smithi051.stdout: "datalog.sync-status.shard.4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc.3", 2020-11-09T18:20:13.444 INFO:teuthology.orchestra.run.smithi051.stdout: "datalog.sync-status.11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:20:13.445 INFO:teuthology.orchestra.run.smithi051.stdout: "datalog.sync-status.shard.11f516a7-3288-49a6-9a89-b45ac1dae91a.1", 2020-11-09T18:20:13.445 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.38-ymzeui-38", 2020-11-09T18:20:13.445 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.15-ymzeui-15", 2020-11-09T18:20:13.445 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4545.1-ymzeui-44", 2020-11-09T18:20:13.445 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.8-ymzeui-8", 2020-11-09T18:20:13.445 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000050", 2020-11-09T18:20:13.446 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000102", 2020-11-09T18:20:13.446 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000124", 2020-11-09T18:20:13.446 INFO:teuthology.orchestra.run.smithi051.stdout: "datalog.sync-status.shard.11f516a7-3288-49a6-9a89-b45ac1dae91a.2", 2020-11-09T18:20:13.446 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000001", 2020-11-09T18:20:13.446 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000126", 2020-11-09T18:20:13.446 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000055", 2020-11-09T18:20:13.447 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000061", 2020-11-09T18:20:13.447 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000097", 2020-11-09T18:20:13.447 INFO:teuthology.orchestra.run.smithi051.stdout: "2020-11-09-18-11f516a7-3288-49a6-9a89-b45ac1dae91a.4347.12-ymzeui-12", 2020-11-09T18:20:13.447 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000042", 2020-11-09T18:20:13.447 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000075", 2020-11-09T18:20:13.447 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000120", 2020-11-09T18:20:13.448 INFO:teuthology.orchestra.run.smithi051.stdout: "obj_delete_at_hint.0000000030", 2020-11-09T18:20:13.448 INFO:teuthology.orchestra.run.smithi051.stdout: "meta.log.a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9.1", 2020-11-09T18:20:13.448 INFO:teuthology.orchestra.run.smithi051.stdout: "data.full-sync.index.4ed44bd6-3ba3-47d6-a46f-3cb9bef974cc.0" 2020-11-09T18:20:13.448 INFO:teuthology.orchestra.run.smithi051.stdout:] 2020-11-09T18:20:13.448 INFO:teuthology.orchestra.run.smithi051.stdout: 2020-11-09T18:20:13.449 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:20:13.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:20:13.450 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:20:13.491 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:20:13.491 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:20:13.750 INFO:teuthology.orchestra.run.smithi078.stdout:[ 2020-11-09T18:20:13.751 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000078", 2020-11-09T18:20:13.751 INFO:teuthology.orchestra.run.smithi078.stdout: "meta.history", 2020-11-09T18:20:13.751 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000070", 2020-11-09T18:20:13.751 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000104", 2020-11-09T18:20:13.751 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000026", 2020-11-09T18:20:13.751 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000028", 2020-11-09T18:20:13.752 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000040", 2020-11-09T18:20:13.752 INFO:teuthology.orchestra.run.smithi078.stdout: "meta.full-sync.index.0", 2020-11-09T18:20:13.752 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000015", 2020-11-09T18:20:13.752 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000069", 2020-11-09T18:20:13.752 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000095", 2020-11-09T18:20:13.752 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000003", 2020-11-09T18:20:13.752 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000047", 2020-11-09T18:20:13.753 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000035", 2020-11-09T18:20:13.753 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000037", 2020-11-09T18:20:13.753 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000024", 2020-11-09T18:20:13.753 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000000", 2020-11-09T18:20:13.753 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000031", 2020-11-09T18:20:13.753 INFO:teuthology.orchestra.run.smithi078.stdout: "data_log.3", 2020-11-09T18:20:13.753 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000076", 2020-11-09T18:20:13.754 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000113", 2020-11-09T18:20:13.754 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000005", 2020-11-09T18:20:13.754 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000011", 2020-11-09T18:20:13.754 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000058", 2020-11-09T18:20:13.754 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000052", 2020-11-09T18:20:13.754 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000088", 2020-11-09T18:20:13.754 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000080", 2020-11-09T18:20:13.755 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000090", 2020-11-09T18:20:13.755 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000110", 2020-11-09T18:20:13.755 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000096", 2020-11-09T18:20:13.755 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000087", 2020-11-09T18:20:13.755 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000008", 2020-11-09T18:20:13.755 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000006", 2020-11-09T18:20:13.755 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000029", 2020-11-09T18:20:13.756 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000089", 2020-11-09T18:20:13.756 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000022", 2020-11-09T18:20:13.756 INFO:teuthology.orchestra.run.smithi078.stdout: "bilog.trim", 2020-11-09T18:20:13.756 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000016", 2020-11-09T18:20:13.756 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000041", 2020-11-09T18:20:13.756 INFO:teuthology.orchestra.run.smithi078.stdout: "data.full-sync.index.11f516a7-3288-49a6-9a89-b45ac1dae91a.3", 2020-11-09T18:20:13.756 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000018", 2020-11-09T18:20:13.757 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000092", 2020-11-09T18:20:13.757 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000014", 2020-11-09T18:20:13.757 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000112", 2020-11-09T18:20:13.757 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000007", 2020-11-09T18:20:13.757 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000021", 2020-11-09T18:20:13.757 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000064", 2020-11-09T18:20:13.757 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000071", 2020-11-09T18:20:13.757 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000074", 2020-11-09T18:20:13.758 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000081", 2020-11-09T18:20:13.758 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000009", 2020-11-09T18:20:13.758 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000121", 2020-11-09T18:20:13.758 INFO:teuthology.orchestra.run.smithi078.stdout: "data_log.1", 2020-11-09T18:20:13.758 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000125", 2020-11-09T18:20:13.758 INFO:teuthology.orchestra.run.smithi078.stdout: "meta.log.bbf9f070-21b9-4940-95e9-d031ba1c647f.0", 2020-11-09T18:20:13.758 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000082", 2020-11-09T18:20:13.759 INFO:teuthology.orchestra.run.smithi078.stdout: "meta.full-sync.index.1", 2020-11-09T18:20:13.759 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000105", 2020-11-09T18:20:13.759 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000059", 2020-11-09T18:20:13.759 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000077", 2020-11-09T18:20:13.759 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000032", 2020-11-09T18:20:13.759 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000053", 2020-11-09T18:20:13.759 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000091", 2020-11-09T18:20:13.760 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000065", 2020-11-09T18:20:13.760 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000083", 2020-11-09T18:20:13.760 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000010", 2020-11-09T18:20:13.760 INFO:teuthology.orchestra.run.smithi078.stdout: "mdlog.sync-status.shard.2", 2020-11-09T18:20:13.760 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000045", 2020-11-09T18:20:13.760 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000002", 2020-11-09T18:20:13.760 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000116", 2020-11-09T18:20:13.761 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000034", 2020-11-09T18:20:13.761 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000101", 2020-11-09T18:20:13.761 INFO:teuthology.orchestra.run.smithi078.stdout: "mdlog.sync-status.shard.1", 2020-11-09T18:20:13.761 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000079", 2020-11-09T18:20:13.761 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000049", 2020-11-09T18:20:13.761 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000117", 2020-11-09T18:20:13.761 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000044", 2020-11-09T18:20:13.761 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000066", 2020-11-09T18:20:13.762 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000068", 2020-11-09T18:20:13.762 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000085", 2020-11-09T18:20:13.762 INFO:teuthology.orchestra.run.smithi078.stdout: "data.full-sync.index.11f516a7-3288-49a6-9a89-b45ac1dae91a.0", 2020-11-09T18:20:13.762 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000073", 2020-11-09T18:20:13.762 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000038", 2020-11-09T18:20:13.762 INFO:teuthology.orchestra.run.smithi078.stdout: "meta.full-sync.index.3", 2020-11-09T18:20:13.762 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000118", 2020-11-09T18:20:13.763 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000036", 2020-11-09T18:20:13.763 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000103", 2020-11-09T18:20:13.763 INFO:teuthology.orchestra.run.smithi078.stdout: "mdlog.sync-status.shard.3", 2020-11-09T18:20:13.763 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000119", 2020-11-09T18:20:13.763 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000098", 2020-11-09T18:20:13.763 INFO:teuthology.orchestra.run.smithi078.stdout: "mdlog.sync-status.shard.0", 2020-11-09T18:20:13.763 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000027", 2020-11-09T18:20:13.764 INFO:teuthology.orchestra.run.smithi078.stdout: "mdlog.sync-status", 2020-11-09T18:20:13.764 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000019", 2020-11-09T18:20:13.764 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000039", 2020-11-09T18:20:13.764 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000100", 2020-11-09T18:20:13.764 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000093", 2020-11-09T18:20:13.764 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000004", 2020-11-09T18:20:13.764 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000063", 2020-11-09T18:20:13.764 INFO:teuthology.orchestra.run.smithi078.stdout: "datalog.sync-status.shard.11f516a7-3288-49a6-9a89-b45ac1dae91a.3", 2020-11-09T18:20:13.765 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000122", 2020-11-09T18:20:13.765 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000057", 2020-11-09T18:20:13.765 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000054", 2020-11-09T18:20:13.765 INFO:teuthology.orchestra.run.smithi078.stdout: "data.full-sync.index.11f516a7-3288-49a6-9a89-b45ac1dae91a.2", 2020-11-09T18:20:13.765 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000114", 2020-11-09T18:20:13.765 INFO:teuthology.orchestra.run.smithi078.stdout: "data_log.0", 2020-11-09T18:20:13.765 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000012", 2020-11-09T18:20:13.766 INFO:teuthology.orchestra.run.smithi078.stdout: "meta.log.a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9.0", 2020-11-09T18:20:13.766 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000084", 2020-11-09T18:20:13.766 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000043", 2020-11-09T18:20:13.766 INFO:teuthology.orchestra.run.smithi078.stdout: "data.full-sync.index.11f516a7-3288-49a6-9a89-b45ac1dae91a.1", 2020-11-09T18:20:13.766 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000111", 2020-11-09T18:20:13.766 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000048", 2020-11-09T18:20:13.766 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000020", 2020-11-09T18:20:13.767 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000099", 2020-11-09T18:20:13.767 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000056", 2020-11-09T18:20:13.767 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000072", 2020-11-09T18:20:13.767 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000062", 2020-11-09T18:20:13.767 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000115", 2020-11-09T18:20:13.767 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000033", 2020-11-09T18:20:13.767 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000025", 2020-11-09T18:20:13.767 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000123", 2020-11-09T18:20:13.768 INFO:teuthology.orchestra.run.smithi078.stdout: "datalog.sync-status.shard.11f516a7-3288-49a6-9a89-b45ac1dae91a.0", 2020-11-09T18:20:13.768 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000108", 2020-11-09T18:20:13.768 INFO:teuthology.orchestra.run.smithi078.stdout: "meta.log.f6258aba-300a-4d5d-ad93-a7c198a0e0a1.0", 2020-11-09T18:20:13.768 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000094", 2020-11-09T18:20:13.768 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000060", 2020-11-09T18:20:13.768 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000109", 2020-11-09T18:20:13.768 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000013", 2020-11-09T18:20:13.769 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000051", 2020-11-09T18:20:13.769 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000106", 2020-11-09T18:20:13.769 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000017", 2020-11-09T18:20:13.769 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000023", 2020-11-09T18:20:13.769 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000107", 2020-11-09T18:20:13.770 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000046", 2020-11-09T18:20:13.770 INFO:teuthology.orchestra.run.smithi078.stdout: "meta.full-sync.index.2", 2020-11-09T18:20:13.770 INFO:teuthology.orchestra.run.smithi078.stdout: "data_log.2", 2020-11-09T18:20:13.770 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000067", 2020-11-09T18:20:13.770 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000086", 2020-11-09T18:20:13.770 INFO:teuthology.orchestra.run.smithi078.stdout: "datalog.sync-status.11f516a7-3288-49a6-9a89-b45ac1dae91a", 2020-11-09T18:20:13.770 INFO:teuthology.orchestra.run.smithi078.stdout: "datalog.sync-status.shard.11f516a7-3288-49a6-9a89-b45ac1dae91a.1", 2020-11-09T18:20:13.771 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000050", 2020-11-09T18:20:13.771 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000102", 2020-11-09T18:20:13.771 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000124", 2020-11-09T18:20:13.771 INFO:teuthology.orchestra.run.smithi078.stdout: "datalog.sync-status.shard.11f516a7-3288-49a6-9a89-b45ac1dae91a.2", 2020-11-09T18:20:13.771 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000001", 2020-11-09T18:20:13.771 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000126", 2020-11-09T18:20:13.771 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000055", 2020-11-09T18:20:13.771 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000061", 2020-11-09T18:20:13.772 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000097", 2020-11-09T18:20:13.772 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000042", 2020-11-09T18:20:13.772 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000075", 2020-11-09T18:20:13.772 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000120", 2020-11-09T18:20:13.772 INFO:teuthology.orchestra.run.smithi078.stdout: "obj_delete_at_hint.0000000030", 2020-11-09T18:20:13.772 INFO:teuthology.orchestra.run.smithi078.stdout: "meta.log.a628cd0f-75ea-4aec-99a0-4bcf5bf2fcd9.1" 2020-11-09T18:20:13.772 INFO:teuthology.orchestra.run.smithi078.stdout:] 2020-11-09T18:20:13.773 INFO:teuthology.orchestra.run.smithi078.stdout: 2020-11-09T18:20:13.863 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:20:13.864 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:20:13.865 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:20:13.905 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:20:13.906 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:20:14.178 INFO:teuthology.orchestra.run.smithi051.stdout:[ 2020-11-09T18:20:14.178 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:20:14.179 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "00000000000000000000:00000000000000002756", 2020-11-09T18:20:14.179 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "2020-11-09T18:20:12.337928Z" 2020-11-09T18:20:14.179 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:20:14.179 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:20:14.179 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "00000000000000000000:00000000000000002761", 2020-11-09T18:20:14.179 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "2020-11-09T18:20:12.341026Z" 2020-11-09T18:20:14.180 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:20:14.180 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:20:14.180 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "00000000000000000000:00000000000000001964", 2020-11-09T18:20:14.180 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "2020-11-09T18:20:12.344471Z" 2020-11-09T18:20:14.180 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:20:14.180 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:20:14.180 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "00000000000000000000:00000000000000002756", 2020-11-09T18:20:14.181 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "2020-11-09T18:20:12.347750Z" 2020-11-09T18:20:14.181 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:20:14.181 INFO:teuthology.orchestra.run.smithi051.stdout:] 2020-11-09T18:20:14.182 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-11-09T18:20:14.183 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:20:14.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:20:14.184 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:20:14.225 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:20:14.225 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:20:14.494 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:20:14.494 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:20:14.494 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:20:14.494 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:20:14.494 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:20:14.495 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:20:14.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:14.495 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:20:14.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:14.495 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:20:14.495 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:14.495 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:20:14.496 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:20:14.496 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:14.496 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:20:14.496 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:14.496 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:20:14.496 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:14.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:14.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:14.497 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:20:14.497 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:14.497 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:20:14.497 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:20:14.497 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:14.497 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:20:14.498 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:14.498 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:20:14.498 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:14.498 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:14.498 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:14.498 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:20:14.498 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:14.499 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:20:14.499 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:20:14.499 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:14.499 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:20:14.499 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:14.499 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:20:14.499 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:14.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:14.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:14.500 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:20:14.500 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:14.500 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:20:14.500 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:20:14.500 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:14.501 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:20:14.501 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:14.501 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:20:14.501 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:14.501 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:14.501 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:20:14.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:14.502 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:20:14.502 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:20:14.502 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:20:14.502 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:14.502 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:20:14.503 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:20:19.505 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:20:19.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:20:19.506 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:20:19.548 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:20:19.549 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:20:19.802 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:20:19.803 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:20:19.803 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:20:19.803 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:20:19.803 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:20:19.803 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:20:19.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:19.804 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:20:19.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:19.804 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:20:19.804 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:19.804 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:20:19.804 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:20:19.805 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:19.805 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:20:19.805 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:19.805 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:20:19.805 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:19.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:19.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:19.806 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:20:19.806 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:19.806 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:20:19.806 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:20:19.806 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:19.806 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:20:19.806 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:19.807 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:20:19.807 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:19.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:19.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:19.807 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:20:19.807 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:19.807 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:20:19.808 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:20:19.808 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:19.808 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:20:19.808 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:19.808 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:20:19.808 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:19.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:19.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:19.809 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:20:19.809 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:19.809 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:20:19.809 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:20:19.809 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:19.809 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:20:19.809 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:19.810 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:20:19.810 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:19.810 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:19.810 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:20:19.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:19.810 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:20:19.811 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:20:19.811 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:20:19.811 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:19.811 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:20:19.812 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:20:24.813 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:20:24.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:20:24.815 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:20:24.861 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:20:24.862 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:20:25.114 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:20:25.115 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:20:25.115 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:20:25.115 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:20:25.115 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:20:25.115 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:20:25.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:25.116 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:20:25.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:25.117 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:20:25.117 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:25.117 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:20:25.117 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:20:25.118 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:25.118 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:20:25.119 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:25.119 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:20:25.119 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:25.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:25.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:25.120 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:20:25.120 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:25.120 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:20:25.121 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:20:25.121 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:25.121 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:20:25.121 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:25.121 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:20:25.122 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:25.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:25.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:25.122 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:20:25.122 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:25.123 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:20:25.123 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:20:25.123 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:25.123 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:20:25.123 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:25.124 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:20:25.124 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:25.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:25.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:25.124 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:20:25.125 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:25.125 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:20:25.125 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:20:25.125 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:25.125 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:20:25.126 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:25.126 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:20:25.126 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:25.126 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:25.126 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:20:25.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:25.127 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:20:25.127 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:20:25.127 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:20:25.127 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:25.128 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:20:25.129 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:20:30.130 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:20:30.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:20:30.131 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:20:30.173 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:20:30.173 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:20:30.436 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:20:30.436 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:20:30.436 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:20:30.437 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:20:30.437 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:20:30.437 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:20:30.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:30.437 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:20:30.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:30.438 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:20:30.438 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:30.438 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:20:30.438 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:20:30.438 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:30.438 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:20:30.438 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:30.439 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:20:30.439 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:30.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:30.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:30.439 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:20:30.439 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:30.439 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:20:30.440 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:20:30.440 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:30.440 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:20:30.440 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:30.440 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:20:30.440 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:30.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:30.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:30.441 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:20:30.441 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:30.441 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:20:30.441 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:20:30.441 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:30.441 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:20:30.441 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:30.442 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:20:30.442 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:30.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:30.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:30.442 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:20:30.442 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:30.443 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:20:30.443 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:20:30.443 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:30.443 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:20:30.443 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:30.443 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:20:30.443 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:30.443 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:30.444 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:20:30.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:30.444 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:20:30.444 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:20:30.444 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:20:30.444 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:30.444 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:20:30.445 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:20:35.446 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:20:35.447 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:20:35.448 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:20:35.489 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:20:35.490 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:20:35.739 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:20:35.740 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:20:35.740 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:20:35.740 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:20:35.740 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:20:35.741 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:20:35.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:35.741 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:20:35.741 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:35.741 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:20:35.741 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:35.742 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:20:35.742 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:20:35.742 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:35.742 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:20:35.742 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:35.743 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:20:35.743 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:35.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:35.743 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:35.743 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:20:35.743 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:35.743 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:20:35.744 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:20:35.744 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:35.744 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:20:35.744 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:35.744 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:20:35.745 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:35.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:35.745 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:35.745 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:20:35.745 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:35.745 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:20:35.746 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:20:35.746 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:35.746 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:20:35.746 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:35.746 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:20:35.746 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:35.746 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:35.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:35.747 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:20:35.747 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:35.747 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:20:35.747 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:20:35.747 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:35.747 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:20:35.747 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:35.748 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:20:35.748 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:35.748 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:35.748 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:20:35.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:35.748 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:20:35.748 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:20:35.749 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:20:35.749 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:35.749 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:20:35.750 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:20:40.751 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:20:40.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:20:40.753 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:20:40.796 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:20:40.797 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:20:41.065 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:20:41.065 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:20:41.066 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:20:41.066 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:20:41.066 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:20:41.066 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:20:41.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:41.066 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:20:41.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:41.067 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:20:41.067 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:41.067 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:20:41.067 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:20:41.067 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:41.067 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:20:41.067 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:41.068 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:20:41.068 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:41.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:41.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:41.068 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:20:41.068 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:41.068 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:20:41.069 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:20:41.069 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:41.069 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:20:41.069 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:41.069 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:20:41.069 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:41.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:41.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:41.070 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:20:41.070 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:41.070 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:20:41.070 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:20:41.070 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:41.070 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:20:41.071 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:41.071 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:20:41.071 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:41.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:41.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:41.071 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:20:41.071 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:41.071 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:20:41.072 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:20:41.072 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:41.072 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:20:41.072 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:41.072 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:20:41.072 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:41.072 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:41.073 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:20:41.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:41.073 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:20:41.073 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:20:41.073 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:20:41.073 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:41.073 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:20:41.074 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:20:46.076 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:20:46.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:20:46.077 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:20:46.119 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:20:46.119 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:20:46.378 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:20:46.378 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:20:46.379 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:20:46.379 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:20:46.379 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:20:46.380 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:20:46.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:46.380 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:20:46.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:46.381 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:20:46.381 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:46.381 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:20:46.382 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:20:46.382 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:46.382 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:20:46.383 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:46.383 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:20:46.383 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:46.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:46.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:46.384 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:20:46.385 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:46.385 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:20:46.385 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:20:46.385 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:46.386 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:20:46.386 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:46.386 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:20:46.387 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:46.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:46.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:46.388 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:20:46.388 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:46.388 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:20:46.389 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:20:46.389 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:46.389 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:20:46.390 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:46.390 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:20:46.390 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:46.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:46.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:46.391 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:20:46.391 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:46.392 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:20:46.392 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:20:46.392 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:46.393 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:20:46.393 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:46.393 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:20:46.393 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:46.394 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:46.394 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:20:46.394 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:46.395 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:20:46.395 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:20:46.395 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:20:46.396 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:46.396 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:20:46.397 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:20:51.399 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:20:51.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:20:51.401 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:20:51.445 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:20:51.446 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:20:51.687 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:20:51.687 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:20:51.687 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:20:51.687 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:20:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:20:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:20:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:20:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:20:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:51.689 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:20:51.689 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:20:51.689 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:51.689 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:20:51.689 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:51.689 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:20:51.689 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:51.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:51.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:51.690 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:20:51.690 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:51.690 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:20:51.690 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:20:51.690 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:20:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:20:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:20:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:20:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:20:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:20:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:20:51.693 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:51.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:51.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:51.693 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:20:51.693 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:51.693 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:20:51.693 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:20:51.694 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:51.694 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:20:51.694 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:51.694 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:20:51.694 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:51.694 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:51.694 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:20:51.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:51.695 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:20:51.695 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:20:51.695 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:20:51.695 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:51.695 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:20:51.696 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:20:56.697 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:20:56.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:20:56.699 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:20:56.739 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:20:56.739 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:20:56.985 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:20:56.985 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:20:56.985 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:20:56.986 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:20:56.986 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:20:56.986 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:20:56.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:56.986 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:20:56.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:56.987 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:20:56.987 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:56.987 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:20:56.987 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:20:56.987 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:56.988 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:20:56.988 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:56.988 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:20:56.988 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:56.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:56.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:56.989 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:20:56.989 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:56.989 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:20:56.989 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:20:56.989 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:56.989 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:20:56.990 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:56.990 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:20:56.990 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:56.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:56.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:56.990 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:20:56.991 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:56.991 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:20:56.991 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:20:56.991 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:56.991 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:20:56.991 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:56.992 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:20:56.992 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:56.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:56.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:20:56.992 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:20:56.992 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:20:56.993 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:20:56.993 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:20:56.993 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:20:56.993 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:20:56.993 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:20:56.993 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:20:56.994 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:56.994 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:56.994 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:20:56.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:20:56.994 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:20:56.994 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:20:56.995 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:20:56.995 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:20:56.995 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:20:56.996 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:21:01.997 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:21:01.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:21:01.999 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:21:02.042 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:21:02.043 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:21:02.317 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:21:02.317 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:21:02.317 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:21:02.318 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:21:02.318 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:21:02.318 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:21:02.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:02.318 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:21:02.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:21:02.319 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:21:02.319 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:21:02.319 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:21:02.319 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:21:02.319 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:21:02.319 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:21:02.319 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:21:02.320 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:21:02.320 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:02.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:02.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:21:02.320 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:21:02.320 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:21:02.320 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:21:02.320 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:21:02.321 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:21:02.321 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:21:02.321 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:21:02.321 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:21:02.321 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:02.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:02.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:21:02.322 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:21:02.322 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:21:02.322 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:21:02.322 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:21:02.322 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:21:02.322 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:21:02.322 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:21:02.323 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:21:02.323 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:02.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:02.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:21:02.323 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:21:02.323 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:21:02.323 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:21:02.324 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:21:02.324 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:21:02.324 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:21:02.324 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:21:02.324 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:21:02.324 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:02.324 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:02.325 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:21:02.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:02.325 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:21:02.325 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:21:02.325 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:21:02.326 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:02.326 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:21:02.327 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:21:07.328 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:21:07.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:21:07.330 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:21:07.374 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:21:07.374 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:21:07.641 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:21:07.641 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:21:07.642 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:21:07.642 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:21:07.642 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:21:07.642 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:21:07.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:07.642 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:21:07.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:21:07.643 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:21:07.643 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:21:07.643 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:21:07.643 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:21:07.643 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:21:07.643 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:21:07.644 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:21:07.644 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:21:07.644 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:07.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:07.644 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:21:07.644 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:21:07.644 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:21:07.644 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:21:07.645 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:21:07.645 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:21:07.645 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:21:07.645 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:21:07.645 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:21:07.645 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:07.645 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:07.646 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:21:07.646 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:21:07.646 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:21:07.646 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:21:07.646 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:21:07.646 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:21:07.646 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:21:07.647 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:21:07.647 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:21:07.647 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:07.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:07.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:21:07.647 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:21:07.647 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:21:07.648 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:21:07.648 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:21:07.648 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:21:07.648 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:21:07.648 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:21:07.648 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:21:07.648 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:07.648 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:07.649 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:21:07.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:07.649 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:21:07.649 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:21:07.649 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:21:07.649 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:07.649 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:21:07.650 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:21:12.652 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:21:12.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:21:12.653 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:21:12.698 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:21:12.699 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:21:12.958 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:21:12.959 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:21:12.959 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:21:12.959 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:21:12.959 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:21:12.959 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:21:12.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:12.960 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:21:12.960 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:21:12.960 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:21:12.960 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:21:12.961 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:21:12.961 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:21:12.961 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:21:12.961 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:21:12.961 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:21:12.961 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:21:12.962 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:12.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:12.962 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:21:12.962 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:21:12.962 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:21:12.962 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:21:12.962 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:21:12.963 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:21:12.963 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:21:12.963 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:21:12.963 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:21:12.963 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:12.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:12.964 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:21:12.964 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:21:12.964 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:21:12.964 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:21:12.964 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:21:12.964 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:21:12.965 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:21:12.965 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:21:12.965 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:21:12.965 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:12.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:12.965 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:21:12.966 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:21:12.966 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:21:12.966 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:21:12.966 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:21:12.967 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:21:12.967 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:21:12.967 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:21:12.967 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:21:12.967 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:12.967 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:12.968 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:21:12.968 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:12.968 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:21:12.968 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:21:12.968 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:21:12.968 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:12.968 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:21:12.969 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:21:17.970 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:21:17.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:21:17.972 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:21:18.013 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:21:18.014 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:21:18.278 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:21:18.278 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:21:18.278 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:21:18.279 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:21:18.279 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:21:18.279 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:21:18.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:18.280 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:21:18.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:21:18.280 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:21:18.280 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:21:18.281 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:21:18.281 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:21:18.281 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:21:18.281 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:21:18.282 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:21:18.282 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:21:18.282 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:18.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:18.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:21:18.283 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:21:18.283 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:21:18.283 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:21:18.284 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:21:18.284 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:21:18.284 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:21:18.284 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:21:18.285 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:21:18.285 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:18.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:18.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:21:18.285 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:21:18.286 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:21:18.286 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:21:18.286 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:21:18.286 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:21:18.287 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:21:18.287 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:21:18.287 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:21:18.287 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:18.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:18.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:21:18.288 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:21:18.288 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:21:18.288 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:21:18.289 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:21:18.289 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:21:18.289 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:21:18.289 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:21:18.290 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:21:18.290 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:18.290 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:18.290 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:21:18.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:18.291 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:21:18.291 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:21:18.291 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:21:18.291 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:18.291 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:21:18.293 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:21:23.295 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:21:23.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:21:23.296 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:21:23.337 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:21:23.337 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:21:23.590 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:21:23.590 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:21:23.590 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:21:23.591 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:21:23.591 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:21:23.591 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:21:23.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:23.591 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:21:23.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:21:23.592 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:21:23.592 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:21:23.592 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:21:23.592 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:21:23.592 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:21:23.592 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:21:23.592 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:21:23.593 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:21:23.593 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:23.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:23.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:21:23.593 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:21:23.593 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:21:23.594 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:21:23.594 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:21:23.594 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:21:23.594 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:21:23.594 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:21:23.594 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:21:23.594 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:23.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:23.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:21:23.595 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:21:23.595 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:21:23.595 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:21:23.595 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:21:23.595 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:21:23.596 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:21:23.596 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:21:23.596 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:21:23.596 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:23.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:23.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:21:23.597 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:21:23.597 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:21:23.597 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:21:23.597 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:21:23.598 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:21:23.598 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:21:23.599 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:21:23.599 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:21:23.599 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:23.599 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:23.599 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:21:23.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:23.600 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:21:23.600 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:21:23.600 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:21:23.600 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:23.600 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:21:23.601 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:21:28.602 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:21:28.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:21:28.603 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:21:28.647 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:21:28.647 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:21:28.898 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:21:28.898 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:21:28.898 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:21:28.898 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:21:28.899 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:21:28.899 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:21:28.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:28.899 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:21:28.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:21:28.899 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:21:28.900 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:21:28.900 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:21:28.900 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:21:28.900 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:21:28.900 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:21:28.900 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:21:28.900 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:21:28.901 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:28.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:28.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:21:28.901 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:21:28.901 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:21:28.901 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:21:28.902 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:21:28.902 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:21:28.902 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:21:28.902 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:21:28.902 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:21:28.902 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:28.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:28.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:21:28.903 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:21:28.903 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:21:28.903 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:21:28.903 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:21:28.903 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:21:28.903 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:21:28.903 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:21:28.904 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:21:28.904 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:28.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:28.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:21:28.904 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:21:28.904 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:21:28.904 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:21:28.905 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:21:28.905 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:21:28.905 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:21:28.905 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:21:28.905 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:21:28.905 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:28.905 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:28.906 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:21:28.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:28.906 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:21:28.906 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:21:28.906 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:21:28.906 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:28.907 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:21:28.908 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:21:33.909 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:21:33.910 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:21:33.910 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:21:33.952 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:21:33.953 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:21:34.230 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:21:34.231 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:21:34.231 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:21:34.232 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:21:34.232 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:21:34.232 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:21:34.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:34.233 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:21:34.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:21:34.233 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:21:34.233 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:21:34.234 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:21:34.234 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:21:34.234 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:21:34.235 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:21:34.235 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:21:34.235 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:21:34.236 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:34.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:34.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:21:34.236 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:21:34.237 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:21:34.237 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:21:34.237 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:21:34.237 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:21:34.238 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:21:34.238 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:21:34.238 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:21:34.238 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:34.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:34.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:21:34.239 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:21:34.239 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:21:34.239 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:21:34.239 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:21:34.239 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:21:34.240 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:21:34.240 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:21:34.240 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:21:34.240 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:34.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:34.241 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:21:34.241 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:21:34.241 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:21:34.241 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:21:34.241 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:21:34.242 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:21:34.242 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:21:34.242 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:21:34.242 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:21:34.242 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:34.242 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:34.243 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:21:34.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:34.243 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:21:34.243 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:21:34.243 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:21:34.243 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:34.244 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:21:34.245 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:21:39.246 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:21:39.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:21:39.247 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:21:39.289 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:21:39.289 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:21:39.560 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:21:39.560 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:21:39.560 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:21:39.560 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:21:39.560 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:21:39.561 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:21:39.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:39.561 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:21:39.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:21:39.561 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:21:39.561 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:21:39.561 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:21:39.562 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:21:39.562 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:21:39.562 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:21:39.562 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:21:39.562 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:21:39.562 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:39.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:39.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:21:39.563 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:21:39.564 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:21:39.564 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:21:39.564 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:21:39.564 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:21:39.564 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:21:39.564 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:21:39.565 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:21:39.565 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:39.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:39.565 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:21:39.565 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:21:39.565 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:21:39.565 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:21:39.566 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:21:39.566 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:21:39.566 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:21:39.566 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:21:39.566 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:21:39.566 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:39.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:39.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:21:39.567 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:21:39.567 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:21:39.567 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:21:39.567 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:21:39.567 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:21:39.567 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:21:39.568 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:21:39.568 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:21:39.568 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:39.568 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:39.568 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:21:39.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:39.569 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:21:39.569 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:21:39.569 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:21:39.569 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:39.569 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:21:39.572 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:21:44.573 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:21:44.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:21:44.574 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:21:44.618 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:21:44.618 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:21:44.866 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:21:44.866 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:21:44.867 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:21:44.867 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:21:44.867 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:21:44.867 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:21:44.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:44.867 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:21:44.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:21:44.868 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:21:44.868 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:21:44.868 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:21:44.868 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:21:44.868 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:21:44.868 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:21:44.868 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:21:44.869 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:21:44.869 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:44.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:44.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:21:44.869 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:21:44.869 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:21:44.869 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:21:44.870 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:21:44.870 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:21:44.870 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:21:44.870 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:21:44.870 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:21:44.870 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:44.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:44.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:21:44.871 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:21:44.871 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:21:44.871 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:21:44.871 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:21:44.871 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:21:44.872 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:21:44.872 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:21:44.872 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:21:44.872 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:44.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:44.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:21:44.872 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:21:44.872 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:21:44.873 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:21:44.873 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:21:44.873 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:21:44.873 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:21:44.873 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:21:44.873 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:21:44.873 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:44.874 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:44.874 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:21:44.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:44.874 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:21:44.874 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:21:44.874 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:21:44.874 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:44.875 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:21:44.876 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:21:49.877 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:21:49.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:21:49.879 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:21:49.921 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:21:49.921 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:21:50.163 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:21:50.164 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:21:50.164 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:21:50.164 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:21:50.165 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:21:50.165 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:21:50.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:50.166 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:21:50.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:21:50.166 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:21:50.167 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:21:50.167 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:21:50.167 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:21:50.167 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:21:50.168 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:21:50.168 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:21:50.168 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:21:50.169 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:50.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:50.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:21:50.169 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:21:50.170 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:21:50.170 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:21:50.170 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:21:50.171 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:21:50.171 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:21:50.171 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:21:50.172 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:21:50.172 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:50.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:50.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:21:50.173 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:21:50.173 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:21:50.173 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:21:50.174 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:21:50.174 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:21:50.174 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:21:50.175 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:21:50.175 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:21:50.175 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:50.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:50.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:21:50.177 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:21:50.177 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:21:50.177 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:21:50.178 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:21:50.178 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:21:50.178 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:21:50.179 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:21:50.179 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:21:50.179 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:50.179 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:50.180 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:21:50.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:50.180 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:21:50.180 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:21:50.180 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:21:50.181 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:50.181 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:21:50.182 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:21:55.184 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:21:55.185 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:21:55.185 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:21:55.228 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:21:55.228 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:21:55.485 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:21:55.486 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:21:55.486 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:21:55.486 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:21:55.486 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:21:55.486 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:21:55.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:55.487 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:21:55.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:21:55.487 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:21:55.487 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:21:55.487 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:21:55.487 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:21:55.488 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:21:55.488 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:21:55.488 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:21:55.488 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:21:55.488 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:55.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:55.489 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:21:55.489 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:21:55.489 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:21:55.489 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:21:55.489 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:21:55.489 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:21:55.489 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:21:55.490 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:21:55.490 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:21:55.490 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:55.490 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:55.490 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:21:55.490 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:21:55.490 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:21:55.491 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:21:55.491 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:21:55.491 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:21:55.491 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:21:55.491 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:21:55.491 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:21:55.491 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:55.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:55.492 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:21:55.492 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:21:55.492 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:21:55.492 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:21:55.492 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:21:55.492 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:21:55.493 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:21:55.493 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:21:55.493 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:21:55.493 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:55.493 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:55.493 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:21:55.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:21:55.494 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:21:55.494 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:21:55.494 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:21:55.494 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:21:55.494 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:21:55.495 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:22:00.497 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:22:00.498 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:22:00.498 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:22:00.541 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:22:00.541 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:22:00.812 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:22:00.812 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:22:00.812 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:22:00.813 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:22:00.813 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:22:00.813 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:22:00.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:00.813 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:22:00.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:00.813 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:22:00.814 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:00.814 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:00.814 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:00.814 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:00.814 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:00.814 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:00.814 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:00.815 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:00.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:00.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:00.815 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:22:00.815 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:00.815 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:00.815 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:00.816 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:00.816 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:00.816 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:00.816 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:00.816 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:00.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:00.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:00.817 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:22:00.817 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:00.817 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:00.817 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:00.817 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:00.817 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:00.817 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:00.818 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:00.818 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:00.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:00.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:00.818 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:22:00.818 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:00.818 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:00.819 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:00.819 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:00.819 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:00.819 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:00.819 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:00.819 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:00.819 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:00.819 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:22:00.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:00.820 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:22:00.820 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:22:00.820 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:22:00.820 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:00.820 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:22:00.821 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:22:05.822 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:22:05.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:22:05.824 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:22:05.869 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:22:05.870 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:22:06.145 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:22:06.146 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:22:06.146 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:22:06.146 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:22:06.146 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:22:06.146 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:22:06.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:06.147 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:22:06.147 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:06.147 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:22:06.147 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:06.147 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:06.148 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:06.148 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:06.148 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:06.148 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:06.148 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:06.148 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:06.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:06.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:06.149 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:22:06.149 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:06.149 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:06.149 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:06.150 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:06.150 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:06.150 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:06.150 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:06.150 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:06.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:06.151 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:06.151 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:22:06.151 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:06.151 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:06.151 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:06.151 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:06.152 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:06.152 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:06.152 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:06.152 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:06.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:06.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:06.153 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:22:06.153 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:06.153 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:06.153 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:06.153 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:06.153 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:06.154 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:06.155 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:06.155 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:06.155 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:06.155 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:22:06.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:06.156 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:22:06.156 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:22:06.156 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:22:06.156 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:06.157 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:22:06.158 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:22:11.159 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:22:11.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:22:11.161 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:22:11.204 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:22:11.204 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:22:11.468 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:22:11.468 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:22:11.468 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:22:11.469 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:22:11.469 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:22:11.469 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:22:11.469 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:11.469 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:22:11.469 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:11.470 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:22:11.470 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:11.470 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:11.470 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:11.470 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:11.471 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:11.471 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:11.471 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:11.471 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:11.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:11.471 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:11.471 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:22:11.472 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:11.472 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:11.472 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:11.472 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:11.472 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:11.472 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:11.473 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:11.473 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:11.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:11.473 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:11.473 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:22:11.473 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:11.474 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:11.474 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:11.474 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:11.474 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:11.474 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:11.474 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:11.475 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:11.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:11.475 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:11.475 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:22:11.475 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:11.475 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:11.475 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:11.476 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:11.476 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:11.476 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:11.476 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:11.476 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:11.476 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:11.477 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:22:11.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:11.477 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:22:11.477 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:22:11.477 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:22:11.477 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:11.478 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:22:11.479 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:22:16.479 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:22:16.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:22:16.481 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:22:16.524 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:22:16.524 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:22:16.769 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:22:16.769 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:22:16.770 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:22:16.770 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:22:16.770 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:22:16.770 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:22:16.770 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:16.770 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:22:16.770 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:16.771 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:22:16.771 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:16.771 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:16.771 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:16.771 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:16.771 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:16.772 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:16.772 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:16.772 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:16.772 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:16.772 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:16.772 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:22:16.772 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:16.773 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:16.773 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:16.773 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:16.773 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:16.773 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:16.773 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:16.773 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:16.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:16.774 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:16.774 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:22:16.774 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:16.774 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:16.774 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:16.774 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:16.774 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:16.775 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:16.775 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:16.775 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:16.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:16.775 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:16.775 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:22:16.775 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:16.776 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:16.776 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:16.776 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:16.776 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:16.776 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:16.776 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:16.776 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:16.777 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:16.777 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:22:16.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:16.777 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:22:16.777 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:22:16.777 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:22:16.777 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:16.777 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:22:16.779 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:22:21.780 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:22:21.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:22:21.782 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:22:21.825 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:22:21.825 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:22:22.079 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:22:22.079 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:22:22.079 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:22:22.080 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:22:22.080 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:22:22.080 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:22:22.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:22.080 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:22:22.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:22.080 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:22:22.081 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:22.081 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:22.081 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:22.081 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:22.081 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:22.081 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:22.081 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:22.082 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:22.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:22.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:22.082 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:22:22.082 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:22.082 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:22.083 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:22.083 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:22.083 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:22.083 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:22.083 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:22.083 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:22.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:22.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:22.084 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:22:22.084 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:22.084 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:22.084 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:22.084 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:22.084 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:22.085 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:22.085 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:22.085 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:22.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:22.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:22.085 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:22:22.085 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:22.086 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:22.086 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:22.086 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:22.086 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:22.086 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:22.086 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:22.086 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:22.087 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:22.087 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:22:22.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:22.087 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:22:22.087 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:22:22.087 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:22:22.087 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:22.087 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:22:22.088 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:22:27.090 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:22:27.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:22:27.091 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:22:27.133 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:22:27.134 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:22:27.390 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:22:27.390 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:22:27.390 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:22:27.390 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:22:27.391 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:22:27.391 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:22:27.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:27.391 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:22:27.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:27.391 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:22:27.392 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:27.392 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:27.392 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:27.392 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:27.392 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:27.393 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:27.393 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:27.393 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:27.393 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:27.393 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:27.393 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:22:27.394 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:27.394 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:27.394 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:27.394 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:27.394 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:27.394 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:27.395 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:27.395 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:27.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:27.395 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:27.395 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:22:27.396 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:27.396 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:27.396 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:27.396 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:27.396 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:27.396 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:27.397 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:27.397 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:27.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:27.397 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:27.397 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:22:27.398 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:27.398 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:27.398 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:27.399 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:27.399 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:27.399 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:27.399 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:27.399 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:27.399 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:27.400 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:22:27.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:27.400 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:22:27.400 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:22:27.400 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:22:27.400 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:27.401 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:22:27.402 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:22:32.403 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:22:32.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:22:32.405 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:22:32.448 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:22:32.448 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:22:32.698 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:22:32.698 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:22:32.699 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:22:32.699 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:22:32.699 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:22:32.700 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:22:32.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:32.700 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:22:32.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:32.700 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:22:32.700 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:32.701 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:32.701 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:32.701 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:32.701 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:32.701 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:32.701 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:32.701 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:32.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:32.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:32.702 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:22:32.702 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:32.702 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:32.702 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:32.703 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:32.703 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:32.703 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:32.703 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:32.703 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:32.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:32.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:32.704 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:22:32.704 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:32.704 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:32.704 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:32.704 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:32.704 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:32.705 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:32.705 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:32.705 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:32.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:32.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:32.706 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:22:32.706 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:32.706 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:32.707 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:32.707 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:32.707 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:32.707 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:32.707 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:32.707 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:32.708 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:32.708 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:22:32.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:32.708 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:22:32.708 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:22:32.708 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:22:32.708 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:32.709 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:22:32.710 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:22:37.711 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:22:37.712 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:22:37.712 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:22:37.757 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:22:37.758 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:22:38.030 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:22:38.030 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:22:38.031 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:22:38.031 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:22:38.031 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:22:38.031 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:22:38.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:38.032 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:22:38.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:38.033 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:22:38.033 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:38.033 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:38.034 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:38.034 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:38.035 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:38.035 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:38.035 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:38.036 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:38.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:38.036 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:38.037 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:22:38.037 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:38.037 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:38.038 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:38.038 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:38.039 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:38.039 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:38.039 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:38.040 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:38.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:38.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:38.041 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:22:38.041 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:38.041 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:38.041 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:38.042 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:38.042 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:38.042 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:38.043 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:38.043 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:38.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:38.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:38.044 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:22:38.044 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:38.045 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:38.045 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:38.045 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:38.046 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:38.046 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:38.046 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:38.047 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:38.047 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:38.048 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:22:38.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:38.048 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:22:38.049 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:22:38.049 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:22:38.049 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:38.050 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:22:38.051 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:22:43.053 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:22:43.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:22:43.054 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:22:43.097 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:22:43.097 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:22:43.364 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:22:43.364 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:22:43.364 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:22:43.364 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:22:43.365 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:22:43.365 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:22:43.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:43.365 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:22:43.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:43.365 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:22:43.365 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:43.366 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:43.366 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:43.366 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:43.366 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:43.366 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:43.366 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:43.366 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:43.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:43.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:43.367 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:22:43.367 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:43.367 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:43.367 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:43.367 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:43.368 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:43.368 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:43.368 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:43.370 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:43.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:43.370 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:43.370 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:22:43.371 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:43.371 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:43.371 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:43.371 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:43.371 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:43.372 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:43.372 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:43.372 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:43.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:43.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:43.373 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:22:43.373 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:43.373 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:43.374 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:43.374 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:43.374 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:43.375 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:43.375 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:43.375 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:43.375 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:43.376 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:22:43.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:43.376 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:22:43.376 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:22:43.377 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:22:43.377 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:43.377 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:22:43.379 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:22:48.380 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:22:48.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:22:48.381 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:22:48.425 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:22:48.425 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:22:48.698 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:22:48.698 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:22:48.698 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:22:48.699 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:22:48.699 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:22:48.699 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:22:48.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:48.699 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:22:48.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:48.699 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:22:48.700 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:48.700 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:48.700 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:48.700 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:48.700 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:48.700 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:48.700 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:48.701 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:48.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:48.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:48.701 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:22:48.701 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:48.701 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:48.702 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:48.702 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:48.702 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:48.702 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:48.702 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:48.702 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:48.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:48.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:48.703 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:22:48.703 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:48.703 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:48.703 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:48.703 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:48.703 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:48.704 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:48.704 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:48.704 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:48.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:48.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:48.704 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:22:48.704 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:48.704 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:48.705 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:48.705 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:48.705 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:48.705 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:48.705 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:48.705 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:48.705 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:48.706 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:22:48.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:48.706 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:22:48.706 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:22:48.706 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:22:48.706 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:48.706 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:22:48.707 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:22:53.708 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:22:53.709 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:22:53.710 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:22:53.752 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:22:53.752 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:22:54.013 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:22:54.013 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:22:54.013 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:22:54.014 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:22:54.014 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:22:54.014 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:22:54.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:54.014 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:22:54.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:54.015 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:22:54.015 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:54.015 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:54.015 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:54.015 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:54.015 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:54.015 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:54.016 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:54.016 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:54.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:54.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:54.016 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:22:54.016 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:54.017 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:54.017 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:54.017 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:54.017 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:54.017 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:54.017 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:54.018 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:54.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:54.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:54.019 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:22:54.019 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:54.019 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:54.020 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:54.020 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:54.020 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:54.020 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:54.020 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:54.020 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:54.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:54.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:54.021 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:22:54.021 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:54.021 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:54.021 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:54.022 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:54.022 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:54.022 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:54.022 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:54.022 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:54.022 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:54.023 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:22:54.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:54.023 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:22:54.023 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:22:54.023 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:22:54.023 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:54.024 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:22:54.025 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:22:59.026 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:22:59.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:22:59.027 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:22:59.068 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:22:59.069 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:22:59.323 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:22:59.323 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:22:59.323 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:22:59.323 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:22:59.324 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:22:59.324 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:22:59.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:59.324 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:22:59.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:59.324 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:22:59.325 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:59.325 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:59.325 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:59.325 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:59.325 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:59.325 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:59.325 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:59.326 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:59.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:59.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:59.326 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:22:59.326 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:59.326 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:59.326 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:59.327 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:59.327 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:59.327 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:59.327 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:59.327 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:59.327 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:59.327 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:59.328 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:22:59.328 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:59.328 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:59.328 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:59.328 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:59.328 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:59.328 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:59.328 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:59.329 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:59.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:59.329 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:22:59.329 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:22:59.329 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:22:59.329 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:22:59.329 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:22:59.330 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:22:59.330 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:22:59.330 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:22:59.330 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:22:59.330 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:59.330 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:59.330 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:22:59.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:22:59.331 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:22:59.331 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:22:59.331 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:22:59.331 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:22:59.331 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:22:59.332 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:23:04.333 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:23:04.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:23:04.334 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:23:04.377 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:23:04.378 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:23:04.639 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:23:04.640 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:23:04.640 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:23:04.640 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:23:04.640 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:23:04.641 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:23:04.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:04.641 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:23:04.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:23:04.642 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:23:04.642 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:23:04.642 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:23:04.642 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:23:04.643 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:23:04.643 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:23:04.643 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:23:04.643 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:23:04.643 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:04.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:04.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:23:04.644 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:23:04.644 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:23:04.644 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:23:04.644 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:23:04.644 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:23:04.644 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:23:04.644 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:23:04.645 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:23:04.645 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:04.645 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:04.645 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:23:04.645 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:23:04.645 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:23:04.646 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:23:04.646 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:23:04.646 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:23:04.646 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:23:04.647 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:23:04.647 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:23:04.647 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:04.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:04.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:23:04.647 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:23:04.647 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:23:04.647 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:23:04.648 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:23:04.648 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:23:04.648 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:23:04.648 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:23:04.648 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:23:04.648 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:04.648 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:04.649 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:23:04.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:04.649 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:23:04.649 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:23:04.649 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:23:04.649 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:04.649 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:23:04.651 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:23:09.652 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:23:09.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:23:09.653 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:23:09.697 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:23:09.697 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:23:09.967 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:23:09.967 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:23:09.968 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:23:09.968 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:23:09.968 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:23:09.968 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:23:09.968 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:09.969 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:23:09.969 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:23:09.969 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:23:09.969 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:23:09.969 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:23:09.969 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:23:09.970 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:23:09.970 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:23:09.970 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:23:09.970 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:23:09.970 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:09.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:09.971 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:23:09.971 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:23:09.971 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:23:09.971 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:23:09.971 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:23:09.971 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:23:09.972 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:23:09.972 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:23:09.972 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:23:09.972 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:09.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:09.972 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:23:09.973 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:23:09.973 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:23:09.973 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:23:09.973 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:23:09.973 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:23:09.973 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:23:09.974 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:23:09.974 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:23:09.974 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:09.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:09.974 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:23:09.974 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:23:09.975 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:23:09.975 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:23:09.975 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:23:09.975 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:23:09.975 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:23:09.975 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:23:09.975 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:23:09.976 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:09.976 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:09.976 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:23:09.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:09.976 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:23:09.976 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:23:09.977 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:23:09.977 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:09.977 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:23:09.978 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:23:14.979 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:23:14.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:23:14.981 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:23:15.023 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:23:15.024 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:23:15.301 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:23:15.301 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:23:15.302 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:23:15.302 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:23:15.302 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:23:15.302 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:23:15.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:15.302 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:23:15.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:23:15.303 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:23:15.303 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:23:15.303 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:23:15.303 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:23:15.303 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:23:15.304 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:23:15.304 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:23:15.304 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:23:15.304 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:15.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:15.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:23:15.304 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:23:15.305 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:23:15.305 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:23:15.305 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:23:15.305 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:23:15.305 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:23:15.305 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:23:15.305 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:23:15.306 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:15.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:15.306 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:23:15.306 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:23:15.306 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:23:15.306 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:23:15.306 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:23:15.307 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:23:15.307 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:23:15.307 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:23:15.307 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:23:15.307 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:15.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:15.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:23:15.308 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:23:15.308 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:23:15.308 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:23:15.308 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:23:15.308 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:23:15.308 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:23:15.308 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:23:15.309 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:23:15.309 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:15.309 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:15.309 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:23:15.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:15.309 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:23:15.309 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:23:15.309 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:23:15.310 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:15.310 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:23:15.311 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:23:20.312 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:23:20.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:23:20.313 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:23:20.355 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:23:20.356 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:23:20.629 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:23:20.629 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:23:20.630 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:23:20.630 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:23:20.630 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:23:20.630 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:23:20.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:20.630 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:23:20.630 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:23:20.631 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:23:20.631 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:23:20.631 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:23:20.631 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:23:20.631 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:23:20.631 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:23:20.631 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:23:20.632 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:23:20.632 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:20.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:20.632 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:23:20.632 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:23:20.632 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:23:20.632 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:23:20.633 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:23:20.633 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:23:20.633 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:23:20.633 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:23:20.633 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:23:20.633 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:20.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:20.634 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:23:20.634 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:23:20.634 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:23:20.634 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:23:20.634 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:23:20.634 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:23:20.635 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:23:20.635 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:23:20.635 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:23:20.635 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:20.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:20.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:23:20.636 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:23:20.636 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:23:20.636 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:23:20.636 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:23:20.636 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:23:20.636 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:23:20.637 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:23:20.637 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:23:20.637 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:20.637 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:20.637 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:23:20.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:20.637 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:23:20.638 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:23:20.638 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:23:20.638 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:20.638 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:23:20.639 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:23:25.640 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:23:26.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:23:26.150 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:23:26.195 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:23:26.196 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:23:26.455 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:23:26.456 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:23:26.456 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:23:26.456 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:23:26.456 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:23:26.456 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:23:26.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:26.457 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:23:26.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:23:26.457 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:23:26.457 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:23:26.457 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:23:26.457 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:23:26.457 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:23:26.458 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:23:26.458 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:23:26.458 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:23:26.458 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:26.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:26.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:23:26.458 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:23:26.459 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:23:26.459 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:23:26.459 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:23:26.459 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:23:26.459 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:23:26.459 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:23:26.460 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:23:26.460 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:26.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:26.460 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:23:26.460 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:23:26.460 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:23:26.460 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:23:26.460 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:23:26.461 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:23:26.461 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:23:26.461 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:23:26.461 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:23:26.461 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:26.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:26.461 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:23:26.462 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:23:26.462 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:23:26.462 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:23:26.462 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:23:26.462 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:23:26.462 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:23:26.462 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:23:26.463 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:23:26.463 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:26.463 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:26.463 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:23:26.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:26.463 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:23:26.463 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:23:26.463 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:23:26.464 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:26.464 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:23:26.465 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:23:31.466 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:23:31.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:23:31.468 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:23:31.513 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:23:31.513 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:23:31.760 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:23:31.761 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:23:31.761 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:23:31.761 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:23:31.761 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:23:31.761 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:23:31.761 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:31.762 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:23:31.762 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:23:31.762 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:23:31.762 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:23:31.762 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:23:31.762 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:23:31.762 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:23:31.763 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:23:31.763 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:23:31.763 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:23:31.763 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:31.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:31.763 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:23:31.763 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:23:31.764 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:23:31.764 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:23:31.764 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:23:31.764 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:23:31.764 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:23:31.764 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:23:31.764 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:23:31.765 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:31.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:31.765 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:23:31.765 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:23:31.765 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:23:31.765 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:23:31.765 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:23:31.765 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:23:31.766 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:23:31.766 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:23:31.766 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:23:31.766 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:31.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:31.766 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:23:31.767 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:23:31.767 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:23:31.767 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:23:31.767 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:23:31.767 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:23:31.767 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:23:31.768 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:23:31.768 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:23:31.768 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:31.768 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:31.768 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:23:31.768 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:31.769 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:23:31.769 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:23:31.770 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:23:31.770 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:31.770 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:23:31.771 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:23:36.773 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:23:36.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:23:36.774 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:23:36.816 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:23:36.816 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:23:37.081 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:23:37.081 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:23:37.082 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:23:37.082 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:23:37.082 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:23:37.082 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:23:37.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:37.083 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:23:37.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:23:37.083 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:23:37.083 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:23:37.083 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:23:37.083 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:23:37.084 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:23:37.084 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:23:37.084 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:23:37.084 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:23:37.084 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:37.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:37.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:23:37.085 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:23:37.085 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:23:37.085 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:23:37.085 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:23:37.085 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:23:37.086 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:23:37.086 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:23:37.086 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:23:37.086 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:37.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:37.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:23:37.087 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:23:37.087 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:23:37.087 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:23:37.087 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:23:37.087 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:23:37.087 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:23:37.087 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:23:37.088 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:23:37.088 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:37.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:37.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:23:37.088 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:23:37.088 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:23:37.089 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:23:37.089 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:23:37.089 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:23:37.089 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:23:37.089 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:23:37.089 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:23:37.090 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:37.090 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:37.090 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:23:37.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:37.090 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:23:37.091 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:23:37.091 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:23:37.091 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:37.091 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:23:37.092 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:23:42.094 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:23:42.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:23:42.095 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:23:42.139 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:23:42.139 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:23:42.414 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:23:42.414 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:23:42.415 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:23:42.415 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:23:42.415 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:23:42.415 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:23:42.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:42.415 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:23:42.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:23:42.416 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:23:42.416 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:23:42.416 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:23:42.416 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:23:42.416 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:23:42.416 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:23:42.416 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:23:42.417 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:23:42.417 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:42.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:42.417 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:23:42.417 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:23:42.417 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:23:42.417 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:23:42.418 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:23:42.418 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:23:42.418 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:23:42.418 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:23:42.418 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:23:42.418 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:42.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:42.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:23:42.419 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:23:42.419 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:23:42.419 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:23:42.419 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:23:42.419 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:23:42.419 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:23:42.420 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:23:42.420 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:23:42.420 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:42.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:42.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:23:42.420 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:23:42.420 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:23:42.420 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:23:42.421 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:23:42.421 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:23:42.421 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:23:42.421 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:23:42.421 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:23:42.421 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:42.421 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:42.422 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:23:42.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:42.422 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:23:42.422 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:23:42.422 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:23:42.422 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:42.422 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:23:42.423 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:23:47.424 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:23:47.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:23:47.427 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:23:47.471 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:23:47.472 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:23:47.743 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:23:47.743 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:23:47.743 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:23:47.743 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:23:47.743 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:23:47.744 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:23:47.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:47.744 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:23:47.744 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:23:47.744 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:23:47.744 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:23:47.744 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:23:47.745 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:23:47.745 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:23:47.745 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:23:47.745 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:23:47.745 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:23:47.745 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:47.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:47.746 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:23:47.746 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:23:47.746 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:23:47.746 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:23:47.746 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:23:47.746 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:23:47.746 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:23:47.747 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:23:47.747 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:23:47.747 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:47.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:47.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:23:47.747 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:23:47.747 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:23:47.748 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:23:47.748 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:23:47.748 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:23:47.748 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:23:47.748 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:23:47.748 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:23:47.748 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:47.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:47.749 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:23:47.749 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:23:47.749 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:23:47.749 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:23:47.749 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:23:47.749 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:23:47.750 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:23:47.750 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:23:47.750 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:23:47.750 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:47.750 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:47.750 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:23:47.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:47.750 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:23:47.751 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:23:47.751 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:23:47.751 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:47.751 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:23:47.751 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:23:52.755 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:23:52.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:23:52.756 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:23:52.799 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:23:52.799 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:23:53.055 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:23:53.055 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:23:53.055 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:23:53.056 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:23:53.056 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:23:53.056 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:23:53.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:53.056 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:23:53.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:23:53.056 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:23:53.057 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:23:53.057 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:23:53.057 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:23:53.057 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:23:53.057 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:23:53.057 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:23:53.058 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:23:53.058 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:53.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:53.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:23:53.058 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:23:53.058 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:23:53.058 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:23:53.059 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:23:53.059 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:23:53.059 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:23:53.059 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:23:53.059 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:23:53.059 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:53.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:53.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:23:53.060 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:23:53.060 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:23:53.060 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:23:53.060 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:23:53.060 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:23:53.060 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:23:53.060 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:23:53.061 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:23:53.061 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:53.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:53.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:23:53.061 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:23:53.061 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:23:53.062 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:23:53.063 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:23:53.063 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:23:53.063 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:23:53.063 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:23:53.063 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:23:53.063 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:53.063 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:53.064 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:23:53.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:53.064 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:23:53.064 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:23:53.064 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:23:53.064 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:53.064 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:23:53.066 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:23:58.067 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:23:58.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:23:58.068 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:23:58.112 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:23:58.112 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:23:58.375 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:23:58.376 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:23:58.376 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:23:58.376 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:23:58.376 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:23:58.376 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:23:58.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:58.377 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:23:58.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:23:58.377 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:23:58.377 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:23:58.377 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:23:58.377 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:23:58.377 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:23:58.378 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:23:58.378 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:23:58.378 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:23:58.378 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:58.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:58.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:23:58.378 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:23:58.379 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:23:58.379 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:23:58.379 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:23:58.379 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:23:58.379 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:23:58.379 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:23:58.379 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:23:58.380 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:58.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:58.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:23:58.380 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:23:58.380 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:23:58.380 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:23:58.380 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:23:58.380 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:23:58.381 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:23:58.381 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:23:58.381 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:23:58.381 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:58.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:58.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:23:58.381 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:23:58.382 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:23:58.382 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:23:58.382 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:23:58.382 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:23:58.382 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:23:58.382 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:23:58.382 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:23:58.383 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:58.383 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:58.383 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:23:58.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:23:58.383 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:23:58.383 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:23:58.383 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:23:58.384 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:23:58.384 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:23:58.385 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:24:03.386 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:24:03.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:24:03.387 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:24:03.432 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:24:03.432 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:24:03.696 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:24:03.696 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:24:03.697 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:24:03.697 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:24:03.697 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:24:03.697 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:24:03.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:03.697 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:24:03.697 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:24:03.698 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:24:03.698 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:24:03.698 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:24:03.698 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:24:03.698 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:24:03.698 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:24:03.699 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:24:03.699 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:24:03.699 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:03.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:03.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:24:03.699 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:24:03.699 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:24:03.700 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:24:03.700 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:24:03.700 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:24:03.700 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:24:03.700 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:24:03.700 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:24:03.700 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:03.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:03.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:24:03.701 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:24:03.701 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:24:03.701 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:24:03.701 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:24:03.701 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:24:03.702 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:24:03.702 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:24:03.702 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:24:03.702 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:03.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:03.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:24:03.702 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:24:03.703 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:24:03.703 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:24:03.703 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:24:03.703 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:24:03.703 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:24:03.703 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:24:03.703 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:24:03.703 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:03.704 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:03.704 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:24:03.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:03.704 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:24:03.704 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:24:03.704 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:24:03.704 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:03.705 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:24:03.706 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:24:08.707 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:24:08.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:24:08.708 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:24:08.750 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:24:08.751 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:24:09.015 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:24:09.016 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:24:09.016 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:24:09.016 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:24:09.016 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:24:09.016 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:24:09.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:09.017 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:24:09.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:24:09.017 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:24:09.017 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:24:09.017 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:24:09.017 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:24:09.018 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:24:09.018 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:24:09.018 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:24:09.018 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:24:09.018 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:09.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:09.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:24:09.019 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:24:09.019 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:24:09.019 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:24:09.019 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:24:09.019 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:24:09.019 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:24:09.019 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:24:09.020 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:24:09.020 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:09.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:09.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:24:09.020 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:24:09.020 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:24:09.020 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:24:09.021 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:24:09.021 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:24:09.021 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:24:09.021 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:24:09.021 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:24:09.021 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:09.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:09.022 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:24:09.022 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:24:09.022 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:24:09.022 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:24:09.022 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:24:09.022 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:24:09.022 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:24:09.023 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:24:09.023 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:24:09.023 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:09.023 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:09.024 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:24:09.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:09.024 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:24:09.024 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:24:09.024 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:24:09.024 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:09.024 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:24:09.027 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:24:14.028 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:24:14.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:24:14.030 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:24:14.073 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:24:14.073 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:24:14.353 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:24:14.353 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:24:14.353 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:24:14.353 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:24:14.354 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:24:14.354 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:24:14.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:14.354 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:24:14.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:24:14.354 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:24:14.355 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:24:14.355 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:24:14.355 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:24:14.355 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:24:14.355 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:24:14.355 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:24:14.355 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:24:14.355 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:14.356 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:14.356 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:24:14.356 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:24:14.356 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:24:14.356 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:24:14.356 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:24:14.356 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:24:14.357 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:24:14.357 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:24:14.357 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:24:14.357 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:14.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:14.357 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:24:14.357 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:24:14.358 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:24:14.358 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:24:14.358 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:24:14.358 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:24:14.358 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:24:14.358 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:24:14.358 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:24:14.359 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:14.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:14.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:24:14.359 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:24:14.359 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:24:14.359 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:24:14.359 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:24:14.360 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:24:14.360 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:24:14.360 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:24:14.360 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:24:14.360 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:14.360 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:14.360 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:24:14.361 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:14.361 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:24:14.361 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:24:14.361 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:24:14.361 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:14.361 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:24:14.362 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:24:19.364 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:24:19.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:24:19.365 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:24:19.408 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:24:19.409 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:24:19.652 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:24:19.653 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:24:19.653 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:24:19.653 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:24:19.653 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:24:19.653 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:24:19.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:19.654 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:24:19.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:24:19.654 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:24:19.654 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:24:19.654 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:24:19.654 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:24:19.654 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:24:19.655 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:24:19.655 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:24:19.655 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:24:19.655 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:19.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:19.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:24:19.655 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:24:19.656 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:24:19.656 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:24:19.656 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:24:19.656 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:24:19.656 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:24:19.656 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:24:19.656 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:24:19.657 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:19.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:19.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:24:19.657 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:24:19.657 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:24:19.657 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:24:19.657 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:24:19.657 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:24:19.658 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:24:19.658 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:24:19.658 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:24:19.658 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:19.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:19.658 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:24:19.659 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:24:19.659 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:24:19.659 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:24:19.659 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:24:19.659 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:24:19.659 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:24:19.659 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:24:19.659 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:24:19.660 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:19.660 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:19.660 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:24:19.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:19.660 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:24:19.660 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:24:19.660 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:24:19.661 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:19.661 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:24:19.662 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:24:24.663 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:24:24.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:24:24.664 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:24:24.712 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:24:24.713 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:24:24.974 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:24:24.974 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:24:24.974 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:24:24.975 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:24:24.975 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:24:24.975 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:24:24.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:24.975 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:24:24.975 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:24:24.975 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:24:24.976 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:24:24.976 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:24:24.976 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:24:24.976 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:24:24.976 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:24:24.976 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:24:24.976 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:24:24.977 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:24.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:24.977 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:24:24.977 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:24:24.977 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:24:24.977 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:24:24.977 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:24:24.978 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:24:24.978 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:24:24.978 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:24:24.978 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:24:24.978 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:24.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:24.978 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:24:24.979 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:24:24.979 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:24:24.979 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:24:24.979 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:24:24.979 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:24:24.979 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:24:24.979 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:24:24.980 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:24:24.980 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:24.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:24.980 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:24:24.980 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:24:24.980 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:24:24.980 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:24:24.980 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:24:24.981 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:24:24.981 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:24:24.981 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:24:24.981 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:24:24.981 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:24.981 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:24.981 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:24:24.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:24.982 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:24:24.982 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:24:24.982 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:24:24.982 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:24.982 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:24:24.983 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:24:29.985 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:24:29.986 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:24:29.986 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:24:30.030 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:24:30.030 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:24:30.294 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:24:30.295 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:24:30.295 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:24:30.295 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:24:30.295 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:24:30.295 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:24:30.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:30.296 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:24:30.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:24:30.296 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:24:30.296 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:24:30.296 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:24:30.296 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:24:30.296 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:24:30.297 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:24:30.297 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:24:30.297 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:24:30.297 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:30.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:30.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:24:30.297 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:24:30.298 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:24:30.298 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:24:30.298 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:24:30.298 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:24:30.298 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:24:30.298 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:24:30.298 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:24:30.299 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:30.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:30.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:24:30.299 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:24:30.299 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:24:30.299 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:24:30.299 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:24:30.300 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:24:30.300 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:24:30.300 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:24:30.300 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:24:30.300 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:30.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:30.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:24:30.301 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:24:30.301 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:24:30.301 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:24:30.301 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:24:30.302 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:24:30.302 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:24:30.302 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:24:30.302 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:24:30.302 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:30.302 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:30.303 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:24:30.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:30.303 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:24:30.303 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:24:30.303 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:24:30.303 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:30.303 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:24:30.304 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:24:35.306 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:24:35.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:24:35.308 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:24:35.352 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:24:35.352 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:24:35.601 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:24:35.602 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:24:35.602 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:24:35.602 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:24:35.602 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:24:35.603 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:24:35.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:35.603 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:24:35.603 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:24:35.603 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:24:35.603 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:24:35.603 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:24:35.604 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:24:35.604 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:24:35.604 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:24:35.604 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:24:35.604 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:24:35.604 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:35.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:35.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:24:35.605 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:24:35.605 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:24:35.605 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:24:35.605 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:24:35.605 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:24:35.605 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:24:35.606 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:24:35.606 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:24:35.606 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:35.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:35.606 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:24:35.606 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:24:35.606 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:24:35.607 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:24:35.607 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:24:35.607 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:24:35.607 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:24:35.607 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:24:35.607 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:24:35.607 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:35.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:35.608 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:24:35.608 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:24:35.608 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:24:35.608 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:24:35.609 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:24:35.609 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:24:35.609 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:24:35.609 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:24:35.609 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:24:35.609 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:35.609 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:35.610 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:24:35.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:35.610 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:24:35.610 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:24:35.610 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:24:35.610 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:35.610 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:24:35.611 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:24:40.613 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:24:40.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:24:40.614 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:24:40.656 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:24:40.656 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:24:40.923 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:24:40.923 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:24:40.923 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:24:40.924 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:24:40.924 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:24:40.924 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:24:40.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:40.924 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:24:40.924 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:24:40.925 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:24:40.925 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:24:40.925 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:24:40.925 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:24:40.925 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:24:40.926 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:24:40.926 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:24:40.926 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:24:40.926 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:40.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:40.927 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:24:40.927 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:24:40.927 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:24:40.927 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:24:40.927 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:24:40.928 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:24:40.928 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:24:40.928 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:24:40.928 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:24:40.928 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:40.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:40.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:24:40.929 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:24:40.929 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:24:40.929 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:24:40.929 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:24:40.929 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:24:40.929 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:24:40.930 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:24:40.930 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:24:40.930 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:40.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:40.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:24:40.930 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:24:40.931 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:24:40.931 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:24:40.931 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:24:40.932 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:24:40.933 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:24:40.933 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:24:40.933 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:24:40.933 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:40.933 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:40.934 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:24:40.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:40.934 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:24:40.934 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:24:40.935 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:24:40.935 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:40.935 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:24:40.935 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:24:45.936 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:24:45.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:24:45.937 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:24:45.981 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:24:45.981 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:24:46.261 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:24:46.261 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:24:46.262 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:24:46.262 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:24:46.262 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:24:46.262 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:24:46.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:46.262 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:24:46.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:24:46.263 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:24:46.263 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:24:46.263 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:24:46.263 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:24:46.263 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:24:46.263 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:24:46.264 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:24:46.264 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:24:46.264 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:46.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:46.264 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:24:46.264 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:24:46.265 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:24:46.265 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:24:46.265 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:24:46.265 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:24:46.265 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:24:46.265 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:24:46.265 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:24:46.266 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:46.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:46.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:24:46.266 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:24:46.266 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:24:46.266 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:24:46.266 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:24:46.267 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:24:46.267 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:24:46.267 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:24:46.267 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:24:46.267 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:46.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:46.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:24:46.268 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:24:46.268 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:24:46.268 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:24:46.268 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:24:46.268 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:24:46.268 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:24:46.269 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:24:46.269 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:24:46.269 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:46.269 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:46.269 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:24:46.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:46.269 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:24:46.270 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:24:46.270 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:24:46.270 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:46.270 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:24:46.271 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:24:51.273 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:24:51.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:24:51.276 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:24:51.318 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:24:51.318 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:24:51.603 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:24:51.604 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:24:51.604 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:24:51.604 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:24:51.604 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:24:51.605 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:24:51.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:51.605 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:24:51.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:24:51.605 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:24:51.605 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:24:51.605 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:24:51.606 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:24:51.606 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:24:51.606 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:24:51.606 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:24:51.606 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:24:51.606 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:51.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:51.607 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:24:51.607 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:24:51.607 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:24:51.607 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:24:51.607 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:24:51.607 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:24:51.608 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:24:51.608 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:24:51.608 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:24:51.608 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:51.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:51.608 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:24:51.609 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:24:51.609 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:24:51.609 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:24:51.609 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:24:51.609 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:24:51.610 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:24:51.610 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:24:51.610 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:24:51.610 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:51.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:51.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:24:51.611 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:24:51.611 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:24:51.611 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:24:51.611 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:24:51.611 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:24:51.611 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:24:51.612 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:24:51.612 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:24:51.612 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:51.612 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:51.612 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:24:51.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:51.612 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:24:51.613 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:24:51.613 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:24:51.613 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:51.613 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:24:51.614 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:24:56.616 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:24:56.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:24:56.617 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:24:56.662 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:24:56.662 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:24:56.926 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:24:56.927 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:24:56.927 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:24:56.927 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:24:56.927 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:24:56.928 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:24:56.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:56.928 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:24:56.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:24:56.928 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:24:56.929 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:24:56.929 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:24:56.929 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:24:56.929 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:24:56.929 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:24:56.930 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:24:56.930 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:24:56.930 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:56.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:56.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:24:56.930 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:24:56.931 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:24:56.931 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:24:56.931 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:24:56.931 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:24:56.931 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:24:56.931 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:24:56.931 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:24:56.932 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:56.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:56.932 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:24:56.932 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:24:56.932 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:24:56.932 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:24:56.933 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:24:56.933 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:24:56.933 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:24:56.933 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:24:56.933 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:24:56.933 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:56.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:56.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:24:56.934 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:24:56.934 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:24:56.934 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:24:56.934 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:24:56.934 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:24:56.935 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:24:56.935 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:24:56.935 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:24:56.935 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:56.935 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:56.935 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:24:56.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:24:56.936 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:24:56.936 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:24:56.936 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:24:56.936 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:24:56.936 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:24:56.937 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:25:01.939 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:01.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:01.940 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:25:01.982 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:25:01.983 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:25:02.246 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:25:02.247 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:25:02.247 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:25:02.247 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:25:02.247 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:25:02.247 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:25:02.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:02.248 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:25:02.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:02.248 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:25:02.248 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:02.248 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:25:02.249 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:25:02.249 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:02.249 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:02.249 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:02.249 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:25:02.249 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:02.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:02.250 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:02.250 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:25:02.250 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:02.250 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:25:02.250 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:25:02.250 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:02.251 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:02.251 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:02.251 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:25:02.251 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:02.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:02.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:02.251 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:25:02.252 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:02.252 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:25:02.252 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:25:02.252 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:02.252 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:02.252 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:02.252 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:25:02.253 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:02.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:02.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:02.253 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:25:02.253 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:02.253 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:25:02.253 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:25:02.254 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:02.254 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:02.254 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:02.254 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:25:02.254 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:02.254 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:02.254 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:25:02.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:02.255 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:25:02.255 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:25:02.255 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:25:02.255 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:02.255 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:25:02.256 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:25:07.257 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:07.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:07.259 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:25:07.302 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:25:07.303 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:25:07.565 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:25:07.565 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:25:07.565 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:25:07.565 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:25:07.565 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:25:07.566 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:25:07.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:07.566 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:25:07.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:07.566 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:25:07.566 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:07.567 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:25:07.567 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:25:07.567 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:07.567 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:07.567 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:07.567 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:25:07.567 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:07.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:07.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:07.568 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:25:07.568 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:07.568 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:25:07.568 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:25:07.568 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:07.568 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:07.569 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:07.569 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:25:07.569 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:07.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:07.569 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:07.569 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:25:07.569 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:07.570 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:25:07.570 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:25:07.570 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:07.570 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:07.570 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:07.570 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:25:07.570 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:07.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:07.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:07.571 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:25:07.571 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:07.571 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:25:07.571 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:25:07.571 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:07.572 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:07.572 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:07.572 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:25:07.572 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:07.572 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:07.572 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:25:07.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:07.573 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:25:07.573 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:25:07.573 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:25:07.573 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:07.573 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:25:07.574 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:25:12.576 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:12.577 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:12.577 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:25:12.620 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:25:12.620 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:25:12.885 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:25:12.886 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:25:12.886 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:25:12.886 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:25:12.886 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:25:12.886 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:25:12.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:12.887 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:25:12.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:12.887 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:25:12.887 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:12.887 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:25:12.887 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:25:12.888 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:12.888 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:12.888 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:12.888 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:25:12.888 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:12.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:12.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:12.889 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:25:12.889 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:12.889 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:25:12.889 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:25:12.889 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:12.889 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:12.889 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:12.890 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:25:12.890 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:12.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:12.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:12.890 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:25:12.890 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:12.890 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:25:12.891 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:25:12.891 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:12.891 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:12.891 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:12.891 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:25:12.891 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:12.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:12.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:12.892 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:25:12.892 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:12.892 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:25:12.892 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:25:12.892 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:12.892 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:12.892 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:12.893 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:25:12.893 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:12.893 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:12.893 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:25:12.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:12.893 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:25:12.893 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:25:12.894 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:25:12.894 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:12.894 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:25:12.895 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:25:17.896 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:17.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:17.897 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:25:17.939 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:25:17.940 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:25:18.213 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:25:18.214 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:25:18.214 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:25:18.214 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:25:18.215 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:25:18.215 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:25:18.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:18.215 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:25:18.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:18.216 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:25:18.216 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:18.216 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:25:18.216 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:25:18.216 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:18.216 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:18.216 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:18.217 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:25:18.217 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:18.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:18.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:18.217 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:25:18.217 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:18.217 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:25:18.217 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:25:18.218 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:18.219 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:18.219 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:18.219 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:25:18.219 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:18.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:18.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:18.220 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:25:18.220 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:18.220 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:25:18.220 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:25:18.220 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:18.221 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:18.221 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:18.221 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:25:18.221 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:18.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:18.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:18.222 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:25:18.222 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:18.222 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:25:18.222 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:25:18.222 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:18.223 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:18.223 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:18.223 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:25:18.223 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:18.223 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:18.223 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:25:18.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:18.224 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:25:18.224 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:25:18.224 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:25:18.224 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:18.224 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:25:18.226 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:25:23.227 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:23.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:23.229 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:25:23.271 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:25:23.271 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:25:23.537 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:25:23.538 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:25:23.538 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:25:23.538 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:25:23.538 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:25:23.538 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:25:23.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:23.539 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:25:23.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:23.539 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:25:23.539 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:23.539 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:25:23.540 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:25:23.540 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:23.540 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:23.540 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:23.540 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:25:23.540 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:23.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:23.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:23.541 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:25:23.541 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:23.541 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:25:23.541 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:25:23.541 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:23.541 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:23.542 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:23.542 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:25:23.542 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:23.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:23.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:23.542 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:25:23.542 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:23.543 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:25:23.543 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:25:23.543 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:23.543 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:23.543 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:23.543 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:25:23.543 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:23.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:23.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:23.544 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:25:23.544 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:23.544 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:25:23.544 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:25:23.544 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:23.545 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:23.545 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:23.545 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:25:23.545 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:23.545 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:23.545 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:25:23.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:23.545 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:25:23.546 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:25:23.546 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:25:23.546 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:23.546 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:25:23.547 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:25:28.549 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:28.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:28.550 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:25:28.594 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:25:28.595 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:25:28.846 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:25:28.846 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:25:28.847 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:25:28.847 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:25:28.847 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:25:28.847 INFO:teuthology.orchestra.run.smithi078.stdout: "instance_id": 11510630185228400339 2020-11-09T18:25:28.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:28.847 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:25:28.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:28.848 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:25:28.848 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:28.848 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:25:28.848 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:25:28.848 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:28.848 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:28.849 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:28.849 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:25:28.849 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:28.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:28.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:28.849 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:25:28.849 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:28.850 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:25:28.850 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:25:28.850 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:28.850 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:28.850 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:28.851 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:25:28.851 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:28.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:28.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:28.851 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:25:28.851 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:28.851 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:25:28.852 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:25:28.852 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:28.852 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:28.852 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:28.852 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:25:28.852 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:28.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:28.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:28.853 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:25:28.853 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:28.853 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "incremental-sync", 2020-11-09T18:25:28.853 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T18:25:28.853 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:28.853 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:28.854 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:28.854 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000" 2020-11-09T18:25:28.854 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:28.854 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:28.854 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:25:28.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:28.854 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:25:28.855 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 0, 2020-11-09T18:25:28.855 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 0 2020-11-09T18:25:28.855 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:28.855 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:25:28.856 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000, shard=1 master=00000000000000000000:00000000000000002761 target=00000000000000000000:00000000000000000000, shard=2 master=00000000000000000000:00000000000000001964 target=00000000000000000000:00000000000000000000, shard=3 master=00000000000000000000:00000000000000002756 target=00000000000000000000:00000000000000000000 2020-11-09T18:25:33.858 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_disable ... FAIL 2020-11-09T18:25:33.859 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ymzeui-59 2020-11-09T18:25:33.909 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ymzeui-60 2020-11-09T18:25:33.988 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ymzeui-61 2020-11-09T18:25:34.058 INFO:tasks.rgw.c1.client.0.smithi051.stdout:Parse error at line 1: 2020-11-09T18:25:34.058 INFO:tasks.rgw.c1.client.0.smithi051.stdout:syntax error 2020-11-09T18:25:34.099 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... ERROR 2020-11-09T18:25:34.101 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ymzeui-62 2020-11-09T18:25:34.136 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ymzeui-63 2020-11-09T18:25:34.197 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ymzeui-64 2020-11-09T18:25:34.297 INFO:tasks.rgw.c1.client.0.smithi051.stdout:Parse error at line 1: 2020-11-09T18:25:34.297 INFO:tasks.rgw.c1.client.0.smithi051.stdout:syntax error 2020-11-09T18:25:34.339 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_disable_enable ... ERROR 2020-11-09T18:25:34.340 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ymzeui-65 2020-11-09T18:25:34.384 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ymzeui-66 2020-11-09T18:25:34.489 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ymzeui-67 2020-11-09T18:25:34.567 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multipart_object_sync ... ERROR 2020-11-09T18:25:34.568 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ymzeui-68 2020-11-09T18:25:34.625 INFO:tasks.rgw.c1.client.0.smithi051.stdout:Parse error at line 1: 2020-11-09T18:25:34.625 INFO:tasks.rgw.c1.client.0.smithi051.stdout:syntax error 2020-11-09T18:25:34.667 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_encrypted_object_sync ... ERROR 2020-11-09T18:25:34.668 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ymzeui-69 2020-11-09T18:25:34.721 INFO:tasks.rgw.c1.client.0.smithi051.stdout:Parse error at line 1: 2020-11-09T18:25:34.721 INFO:tasks.rgw.c1.client.0.smithi051.stdout:syntax error 2020-11-09T18:25:34.763 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_index_log_trim ... ERROR 2020-11-09T18:25:34.799 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_creation_time ... ok 2020-11-09T18:25:34.800 INFO:tasks.rgw_multisite_tests: 2020-11-09T18:25:34.800 INFO:tasks.rgw_multisite_tests:====================================================================== 2020-11-09T18:25:34.800 INFO:tasks.rgw_multisite_tests:ERROR: tasks.rgw_multi.tests.test_object_sync 2020-11-09T18:25:34.800 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-11-09T18:25:34.801 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2020-11-09T18:25:34.801 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2020-11-09T18:25:34.801 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2020-11-09T18:25:34.801 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-dang-webdir-empty-object/qa/tasks/rgw_multi/tests.py", line 587, in test_object_sync 2020-11-09T18:25:34.802 INFO:tasks.rgw_multisite_tests: k.set_contents_from_string(content) 2020-11-09T18:25:34.802 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 1426, in set_contents_from_string 2020-11-09T18:25:34.802 INFO:tasks.rgw_multisite_tests: encrypt_key=encrypt_key) 2020-11-09T18:25:34.802 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file 2020-11-09T18:25:34.803 INFO:tasks.rgw_multisite_tests: chunked_transfer=chunked_transfer, size=size) 2020-11-09T18:25:34.803 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 750, in send_file 2020-11-09T18:25:34.803 INFO:tasks.rgw_multisite_tests: chunked_transfer=chunked_transfer, size=size) 2020-11-09T18:25:34.803 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 951, in _send_file_internal 2020-11-09T18:25:34.804 INFO:tasks.rgw_multisite_tests: query_args=query_args 2020-11-09T18:25:34.804 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/connection.py", line 668, in make_request 2020-11-09T18:25:34.804 INFO:tasks.rgw_multisite_tests: retry_handler=retry_handler 2020-11-09T18:25:34.804 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/connection.py", line 1071, in make_request 2020-11-09T18:25:34.805 INFO:tasks.rgw_multisite_tests: retry_handler=retry_handler) 2020-11-09T18:25:34.805 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/connection.py", line 940, in _mexe 2020-11-09T18:25:34.805 INFO:tasks.rgw_multisite_tests: request.body, request.headers) 2020-11-09T18:25:34.805 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 884, in sender 2020-11-09T18:25:34.805 INFO:tasks.rgw_multisite_tests: response.status, response.reason, body) 2020-11-09T18:25:34.807 INFO:tasks.rgw_multisite_tests:boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request 2020-11-09T18:25:34.807 INFO:tasks.rgw_multisite_tests:InvalidArgumentymzeui-10tx000000000000000000124-005fa986b4-10f7-test-zone110f7-test-zone1-test-zonegroup 2020-11-09T18:25:34.808 INFO:tasks.rgw_multisite_tests: 2020-11-09T18:25:34.808 INFO:tasks.rgw_multisite_tests:====================================================================== 2020-11-09T18:25:34.809 INFO:tasks.rgw_multisite_tests:ERROR: tasks.rgw_multi.tests.test_object_delete 2020-11-09T18:25:34.809 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-11-09T18:25:34.810 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2020-11-09T18:25:34.810 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2020-11-09T18:25:34.811 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2020-11-09T18:25:34.811 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-dang-webdir-empty-object/qa/tasks/rgw_multi/tests.py", line 610, in test_object_delete 2020-11-09T18:25:34.811 INFO:tasks.rgw_multisite_tests: k.set_contents_from_string(content) 2020-11-09T18:25:34.812 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 1426, in set_contents_from_string 2020-11-09T18:25:34.812 INFO:tasks.rgw_multisite_tests: encrypt_key=encrypt_key) 2020-11-09T18:25:34.813 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file 2020-11-09T18:25:34.813 INFO:tasks.rgw_multisite_tests: chunked_transfer=chunked_transfer, size=size) 2020-11-09T18:25:34.814 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 750, in send_file 2020-11-09T18:25:34.814 INFO:tasks.rgw_multisite_tests: chunked_transfer=chunked_transfer, size=size) 2020-11-09T18:25:34.814 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 951, in _send_file_internal 2020-11-09T18:25:34.815 INFO:tasks.rgw_multisite_tests: query_args=query_args 2020-11-09T18:25:34.815 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/connection.py", line 668, in make_request 2020-11-09T18:25:34.815 INFO:tasks.rgw_multisite_tests: retry_handler=retry_handler 2020-11-09T18:25:34.815 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/connection.py", line 1071, in make_request 2020-11-09T18:25:34.816 INFO:tasks.rgw_multisite_tests: retry_handler=retry_handler) 2020-11-09T18:25:34.816 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/connection.py", line 940, in _mexe 2020-11-09T18:25:34.816 INFO:tasks.rgw_multisite_tests: request.body, request.headers) 2020-11-09T18:25:34.816 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 884, in sender 2020-11-09T18:25:34.817 INFO:tasks.rgw_multisite_tests: response.status, response.reason, body) 2020-11-09T18:25:34.818 INFO:tasks.rgw_multisite_tests:boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request 2020-11-09T18:25:34.818 INFO:tasks.rgw_multisite_tests:InvalidArgumentymzeui-13tx000000000000000000144-005fa986b5-10f7-test-zone110f7-test-zone1-test-zonegroup 2020-11-09T18:25:34.818 INFO:tasks.rgw_multisite_tests: 2020-11-09T18:25:34.818 INFO:tasks.rgw_multisite_tests:====================================================================== 2020-11-09T18:25:34.819 INFO:tasks.rgw_multisite_tests:ERROR: tasks.rgw_multi.tests.test_versioned_object_incremental_sync 2020-11-09T18:25:34.819 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-11-09T18:25:34.819 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2020-11-09T18:25:34.819 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2020-11-09T18:25:34.820 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2020-11-09T18:25:34.820 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-dang-webdir-empty-object/qa/tasks/rgw_multi/tests.py", line 654, in test_versioned_object_incremental_sync 2020-11-09T18:25:34.820 INFO:tasks.rgw_multisite_tests: new_key(source_conn, bucket, 'dummy').set_contents_from_string('') 2020-11-09T18:25:34.820 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 1426, in set_contents_from_string 2020-11-09T18:25:34.821 INFO:tasks.rgw_multisite_tests: encrypt_key=encrypt_key) 2020-11-09T18:25:34.821 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file 2020-11-09T18:25:34.821 INFO:tasks.rgw_multisite_tests: chunked_transfer=chunked_transfer, size=size) 2020-11-09T18:25:34.821 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 750, in send_file 2020-11-09T18:25:34.821 INFO:tasks.rgw_multisite_tests: chunked_transfer=chunked_transfer, size=size) 2020-11-09T18:25:34.822 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 951, in _send_file_internal 2020-11-09T18:25:34.822 INFO:tasks.rgw_multisite_tests: query_args=query_args 2020-11-09T18:25:34.822 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/connection.py", line 668, in make_request 2020-11-09T18:25:34.822 INFO:tasks.rgw_multisite_tests: retry_handler=retry_handler 2020-11-09T18:25:34.823 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/connection.py", line 1071, in make_request 2020-11-09T18:25:34.823 INFO:tasks.rgw_multisite_tests: retry_handler=retry_handler) 2020-11-09T18:25:34.823 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/connection.py", line 940, in _mexe 2020-11-09T18:25:34.823 INFO:tasks.rgw_multisite_tests: request.body, request.headers) 2020-11-09T18:25:34.824 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 882, in sender 2020-11-09T18:25:34.824 INFO:tasks.rgw_multisite_tests: if not self.should_retry(response, chunked_transfer): 2020-11-09T18:25:34.824 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 983, in should_retry 2020-11-09T18:25:34.824 INFO:tasks.rgw_multisite_tests: '%s vs. %s' % (self.etag, self.md5)) 2020-11-09T18:25:34.824 INFO:tasks.rgw_multisite_tests:boto.exception.S3DataError: BotoClientError: ETag from S3 did not match computed MD5. None vs. b'd41d8cd98f00b204e9800998ecf8427e' 2020-11-09T18:25:34.825 INFO:tasks.rgw_multisite_tests: 2020-11-09T18:25:34.825 INFO:tasks.rgw_multisite_tests:====================================================================== 2020-11-09T18:25:34.825 INFO:tasks.rgw_multisite_tests:ERROR: tasks.rgw_multi.tests.test_concurrent_versioned_object_incremental_sync 2020-11-09T18:25:34.825 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-11-09T18:25:34.826 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2020-11-09T18:25:34.826 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2020-11-09T18:25:34.826 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2020-11-09T18:25:34.826 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-dang-webdir-empty-object/qa/tasks/rgw_multi/tests.py", line 709, in test_concurrent_versioned_object_incremental_sync 2020-11-09T18:25:34.827 INFO:tasks.rgw_multisite_tests: new_key(zone, bucket, 'dummy').set_contents_from_string('') 2020-11-09T18:25:34.827 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 1426, in set_contents_from_string 2020-11-09T18:25:34.827 INFO:tasks.rgw_multisite_tests: encrypt_key=encrypt_key) 2020-11-09T18:25:34.827 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file 2020-11-09T18:25:34.828 INFO:tasks.rgw_multisite_tests: chunked_transfer=chunked_transfer, size=size) 2020-11-09T18:25:34.828 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 750, in send_file 2020-11-09T18:25:34.828 INFO:tasks.rgw_multisite_tests: chunked_transfer=chunked_transfer, size=size) 2020-11-09T18:25:34.828 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 951, in _send_file_internal 2020-11-09T18:25:34.829 INFO:tasks.rgw_multisite_tests: query_args=query_args 2020-11-09T18:25:34.829 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/connection.py", line 668, in make_request 2020-11-09T18:25:34.829 INFO:tasks.rgw_multisite_tests: retry_handler=retry_handler 2020-11-09T18:25:34.829 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/connection.py", line 1071, in make_request 2020-11-09T18:25:34.829 INFO:tasks.rgw_multisite_tests: retry_handler=retry_handler) 2020-11-09T18:25:34.830 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/connection.py", line 940, in _mexe 2020-11-09T18:25:34.830 INFO:tasks.rgw_multisite_tests: request.body, request.headers) 2020-11-09T18:25:34.830 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 882, in sender 2020-11-09T18:25:34.830 INFO:tasks.rgw_multisite_tests: if not self.should_retry(response, chunked_transfer): 2020-11-09T18:25:34.831 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 983, in should_retry 2020-11-09T18:25:34.831 INFO:tasks.rgw_multisite_tests: '%s vs. %s' % (self.etag, self.md5)) 2020-11-09T18:25:34.831 INFO:tasks.rgw_multisite_tests:boto.exception.S3DataError: BotoClientError: ETag from S3 did not match computed MD5. None vs. b'd41d8cd98f00b204e9800998ecf8427e' 2020-11-09T18:25:34.831 INFO:tasks.rgw_multisite_tests: 2020-11-09T18:25:34.832 INFO:tasks.rgw_multisite_tests:====================================================================== 2020-11-09T18:25:34.832 INFO:tasks.rgw_multisite_tests:ERROR: tasks.rgw_multi.tests.test_version_suspended_incremental_sync 2020-11-09T18:25:34.832 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-11-09T18:25:34.832 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2020-11-09T18:25:34.833 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2020-11-09T18:25:34.833 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2020-11-09T18:25:34.833 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-dang-webdir-empty-object/qa/tasks/rgw_multi/tests.py", line 736, in test_version_suspended_incremental_sync 2020-11-09T18:25:34.833 INFO:tasks.rgw_multisite_tests: key1.set_contents_from_string('') 2020-11-09T18:25:34.834 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 1426, in set_contents_from_string 2020-11-09T18:25:34.834 INFO:tasks.rgw_multisite_tests: encrypt_key=encrypt_key) 2020-11-09T18:25:34.834 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file 2020-11-09T18:25:34.834 INFO:tasks.rgw_multisite_tests: chunked_transfer=chunked_transfer, size=size) 2020-11-09T18:25:34.834 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 750, in send_file 2020-11-09T18:25:34.835 INFO:tasks.rgw_multisite_tests: chunked_transfer=chunked_transfer, size=size) 2020-11-09T18:25:34.835 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 951, in _send_file_internal 2020-11-09T18:25:34.835 INFO:tasks.rgw_multisite_tests: query_args=query_args 2020-11-09T18:25:34.835 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/connection.py", line 668, in make_request 2020-11-09T18:25:34.836 INFO:tasks.rgw_multisite_tests: retry_handler=retry_handler 2020-11-09T18:25:34.836 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/connection.py", line 1071, in make_request 2020-11-09T18:25:34.836 INFO:tasks.rgw_multisite_tests: retry_handler=retry_handler) 2020-11-09T18:25:34.836 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/connection.py", line 940, in _mexe 2020-11-09T18:25:34.836 INFO:tasks.rgw_multisite_tests: request.body, request.headers) 2020-11-09T18:25:34.837 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 882, in sender 2020-11-09T18:25:34.837 INFO:tasks.rgw_multisite_tests: if not self.should_retry(response, chunked_transfer): 2020-11-09T18:25:34.837 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 983, in should_retry 2020-11-09T18:25:34.837 INFO:tasks.rgw_multisite_tests: '%s vs. %s' % (self.etag, self.md5)) 2020-11-09T18:25:34.838 INFO:tasks.rgw_multisite_tests:boto.exception.S3DataError: BotoClientError: ETag from S3 did not match computed MD5. None vs. b'd41d8cd98f00b204e9800998ecf8427e' 2020-11-09T18:25:34.838 INFO:tasks.rgw_multisite_tests: 2020-11-09T18:25:34.838 INFO:tasks.rgw_multisite_tests:====================================================================== 2020-11-09T18:25:34.838 INFO:tasks.rgw_multisite_tests:ERROR: tasks.rgw_multi.tests.test_delete_marker_full_sync 2020-11-09T18:25:34.839 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-11-09T18:25:34.839 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2020-11-09T18:25:34.839 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2020-11-09T18:25:34.839 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2020-11-09T18:25:34.840 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-dang-webdir-empty-object/qa/tasks/rgw_multi/tests.py", line 773, in test_delete_marker_full_sync 2020-11-09T18:25:34.840 INFO:tasks.rgw_multisite_tests: key1.set_contents_from_string('') 2020-11-09T18:25:34.840 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 1426, in set_contents_from_string 2020-11-09T18:25:34.840 INFO:tasks.rgw_multisite_tests: encrypt_key=encrypt_key) 2020-11-09T18:25:34.840 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file 2020-11-09T18:25:34.841 INFO:tasks.rgw_multisite_tests: chunked_transfer=chunked_transfer, size=size) 2020-11-09T18:25:34.841 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 750, in send_file 2020-11-09T18:25:34.841 INFO:tasks.rgw_multisite_tests: chunked_transfer=chunked_transfer, size=size) 2020-11-09T18:25:34.841 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 951, in _send_file_internal 2020-11-09T18:25:34.842 INFO:tasks.rgw_multisite_tests: query_args=query_args 2020-11-09T18:25:34.842 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/connection.py", line 668, in make_request 2020-11-09T18:25:34.842 INFO:tasks.rgw_multisite_tests: retry_handler=retry_handler 2020-11-09T18:25:34.842 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/connection.py", line 1071, in make_request 2020-11-09T18:25:34.843 INFO:tasks.rgw_multisite_tests: retry_handler=retry_handler) 2020-11-09T18:25:34.843 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/connection.py", line 940, in _mexe 2020-11-09T18:25:34.843 INFO:tasks.rgw_multisite_tests: request.body, request.headers) 2020-11-09T18:25:34.843 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 882, in sender 2020-11-09T18:25:34.843 INFO:tasks.rgw_multisite_tests: if not self.should_retry(response, chunked_transfer): 2020-11-09T18:25:34.844 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 983, in should_retry 2020-11-09T18:25:34.844 INFO:tasks.rgw_multisite_tests: '%s vs. %s' % (self.etag, self.md5)) 2020-11-09T18:25:34.844 INFO:tasks.rgw_multisite_tests:boto.exception.S3DataError: BotoClientError: ETag from S3 did not match computed MD5. None vs. b'd41d8cd98f00b204e9800998ecf8427e' 2020-11-09T18:25:34.844 INFO:tasks.rgw_multisite_tests: 2020-11-09T18:25:34.845 INFO:tasks.rgw_multisite_tests:====================================================================== 2020-11-09T18:25:34.845 INFO:tasks.rgw_multisite_tests:ERROR: tasks.rgw_multi.tests.test_suspended_delete_marker_full_sync 2020-11-09T18:25:34.845 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-11-09T18:25:34.845 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2020-11-09T18:25:34.846 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2020-11-09T18:25:34.846 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2020-11-09T18:25:34.846 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-dang-webdir-empty-object/qa/tasks/rgw_multi/tests.py", line 797, in test_suspended_delete_marker_full_sync 2020-11-09T18:25:34.846 INFO:tasks.rgw_multisite_tests: key1.set_contents_from_string('') 2020-11-09T18:25:34.847 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 1426, in set_contents_from_string 2020-11-09T18:25:34.847 INFO:tasks.rgw_multisite_tests: encrypt_key=encrypt_key) 2020-11-09T18:25:34.847 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file 2020-11-09T18:25:34.847 INFO:tasks.rgw_multisite_tests: chunked_transfer=chunked_transfer, size=size) 2020-11-09T18:25:34.847 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 750, in send_file 2020-11-09T18:25:34.848 INFO:tasks.rgw_multisite_tests: chunked_transfer=chunked_transfer, size=size) 2020-11-09T18:25:34.848 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 951, in _send_file_internal 2020-11-09T18:25:34.848 INFO:tasks.rgw_multisite_tests: query_args=query_args 2020-11-09T18:25:34.848 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/connection.py", line 668, in make_request 2020-11-09T18:25:34.849 INFO:tasks.rgw_multisite_tests: retry_handler=retry_handler 2020-11-09T18:25:34.849 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/connection.py", line 1071, in make_request 2020-11-09T18:25:34.849 INFO:tasks.rgw_multisite_tests: retry_handler=retry_handler) 2020-11-09T18:25:34.849 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/connection.py", line 940, in _mexe 2020-11-09T18:25:34.850 INFO:tasks.rgw_multisite_tests: request.body, request.headers) 2020-11-09T18:25:34.850 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 882, in sender 2020-11-09T18:25:34.850 INFO:tasks.rgw_multisite_tests: if not self.should_retry(response, chunked_transfer): 2020-11-09T18:25:34.850 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 983, in should_retry 2020-11-09T18:25:34.850 INFO:tasks.rgw_multisite_tests: '%s vs. %s' % (self.etag, self.md5)) 2020-11-09T18:25:34.851 INFO:tasks.rgw_multisite_tests:boto.exception.S3DataError: BotoClientError: ETag from S3 did not match computed MD5. None vs. b'd41d8cd98f00b204e9800998ecf8427e' 2020-11-09T18:25:34.851 INFO:tasks.rgw_multisite_tests: 2020-11-09T18:25:34.851 INFO:tasks.rgw_multisite_tests:====================================================================== 2020-11-09T18:25:34.851 INFO:tasks.rgw_multisite_tests:ERROR: tasks.rgw_multi.tests.test_bucket_delete_notempty 2020-11-09T18:25:34.852 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-11-09T18:25:34.852 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2020-11-09T18:25:34.852 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2020-11-09T18:25:34.852 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2020-11-09T18:25:34.853 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-dang-webdir-empty-object/qa/tasks/rgw_multi/tests.py", line 841, in test_bucket_delete_notempty 2020-11-09T18:25:34.853 INFO:tasks.rgw_multisite_tests: k.set_contents_from_string('bar') 2020-11-09T18:25:34.853 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 1426, in set_contents_from_string 2020-11-09T18:25:34.853 INFO:tasks.rgw_multisite_tests: encrypt_key=encrypt_key) 2020-11-09T18:25:34.854 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file 2020-11-09T18:25:34.854 INFO:tasks.rgw_multisite_tests: chunked_transfer=chunked_transfer, size=size) 2020-11-09T18:25:34.854 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 750, in send_file 2020-11-09T18:25:34.854 INFO:tasks.rgw_multisite_tests: chunked_transfer=chunked_transfer, size=size) 2020-11-09T18:25:34.854 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 951, in _send_file_internal 2020-11-09T18:25:34.855 INFO:tasks.rgw_multisite_tests: query_args=query_args 2020-11-09T18:25:34.855 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/connection.py", line 668, in make_request 2020-11-09T18:25:34.855 INFO:tasks.rgw_multisite_tests: retry_handler=retry_handler 2020-11-09T18:25:34.855 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/connection.py", line 1071, in make_request 2020-11-09T18:25:34.856 INFO:tasks.rgw_multisite_tests: retry_handler=retry_handler) 2020-11-09T18:25:34.856 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/connection.py", line 940, in _mexe 2020-11-09T18:25:34.856 INFO:tasks.rgw_multisite_tests: request.body, request.headers) 2020-11-09T18:25:34.856 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 884, in sender 2020-11-09T18:25:34.857 INFO:tasks.rgw_multisite_tests: response.status, response.reason, body) 2020-11-09T18:25:34.857 INFO:tasks.rgw_multisite_tests:boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request 2020-11-09T18:25:34.857 INFO:tasks.rgw_multisite_tests:InvalidArgumentymzeui-36tx0000000000000000002e0-005fa986cb-10f7-test-zone110f7-test-zone1-test-zonegroup 2020-11-09T18:25:34.857 INFO:tasks.rgw_multisite_tests: 2020-11-09T18:25:34.857 INFO:tasks.rgw_multisite_tests:====================================================================== 2020-11-09T18:25:34.858 INFO:tasks.rgw_multisite_tests:ERROR: tasks.rgw_multi.tests.test_datalog_autotrim 2020-11-09T18:25:34.858 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-11-09T18:25:34.858 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2020-11-09T18:25:34.858 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2020-11-09T18:25:34.859 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2020-11-09T18:25:34.859 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-dang-webdir-empty-object/qa/tasks/rgw_multi/tests.py", line 949, in test_datalog_autotrim 2020-11-09T18:25:34.859 INFO:tasks.rgw_multisite_tests: k.set_contents_from_string('body') 2020-11-09T18:25:34.859 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 1426, in set_contents_from_string 2020-11-09T18:25:34.860 INFO:tasks.rgw_multisite_tests: encrypt_key=encrypt_key) 2020-11-09T18:25:34.860 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file 2020-11-09T18:25:34.860 INFO:tasks.rgw_multisite_tests: chunked_transfer=chunked_transfer, size=size) 2020-11-09T18:25:34.860 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 750, in send_file 2020-11-09T18:25:34.861 INFO:tasks.rgw_multisite_tests: chunked_transfer=chunked_transfer, size=size) 2020-11-09T18:25:34.861 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 951, in _send_file_internal 2020-11-09T18:25:34.861 INFO:tasks.rgw_multisite_tests: query_args=query_args 2020-11-09T18:25:34.861 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/connection.py", line 668, in make_request 2020-11-09T18:25:34.861 INFO:tasks.rgw_multisite_tests: retry_handler=retry_handler 2020-11-09T18:25:34.862 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/connection.py", line 1071, in make_request 2020-11-09T18:25:34.862 INFO:tasks.rgw_multisite_tests: retry_handler=retry_handler) 2020-11-09T18:25:34.862 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/connection.py", line 940, in _mexe 2020-11-09T18:25:34.862 INFO:tasks.rgw_multisite_tests: request.body, request.headers) 2020-11-09T18:25:34.863 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 884, in sender 2020-11-09T18:25:34.863 INFO:tasks.rgw_multisite_tests: response.status, response.reason, body) 2020-11-09T18:25:34.863 INFO:tasks.rgw_multisite_tests:boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request 2020-11-09T18:25:34.863 INFO:tasks.rgw_multisite_tests:InvalidArgumentymzeui-46tx0000000000000000000de-005fa98763-11bf-test-zone111bf-test-zone1-test-zonegroup 2020-11-09T18:25:34.864 INFO:tasks.rgw_multisite_tests: 2020-11-09T18:25:34.864 INFO:tasks.rgw_multisite_tests:====================================================================== 2020-11-09T18:25:34.864 INFO:tasks.rgw_multisite_tests:ERROR: tasks.rgw_multi.tests.test_multi_zone_redirect 2020-11-09T18:25:34.864 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-11-09T18:25:34.865 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2020-11-09T18:25:34.865 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2020-11-09T18:25:34.865 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2020-11-09T18:25:34.865 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-dang-webdir-empty-object/qa/tasks/rgw_multi/tests.py", line 993, in test_multi_zone_redirect 2020-11-09T18:25:34.865 INFO:tasks.rgw_multisite_tests: key.set_contents_from_string(data) 2020-11-09T18:25:34.866 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 1426, in set_contents_from_string 2020-11-09T18:25:34.866 INFO:tasks.rgw_multisite_tests: encrypt_key=encrypt_key) 2020-11-09T18:25:34.866 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file 2020-11-09T18:25:34.866 INFO:tasks.rgw_multisite_tests: chunked_transfer=chunked_transfer, size=size) 2020-11-09T18:25:34.867 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 750, in send_file 2020-11-09T18:25:34.867 INFO:tasks.rgw_multisite_tests: chunked_transfer=chunked_transfer, size=size) 2020-11-09T18:25:34.867 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 951, in _send_file_internal 2020-11-09T18:25:34.867 INFO:tasks.rgw_multisite_tests: query_args=query_args 2020-11-09T18:25:34.868 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/connection.py", line 668, in make_request 2020-11-09T18:25:34.868 INFO:tasks.rgw_multisite_tests: retry_handler=retry_handler 2020-11-09T18:25:34.868 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/connection.py", line 1071, in make_request 2020-11-09T18:25:34.868 INFO:tasks.rgw_multisite_tests: retry_handler=retry_handler) 2020-11-09T18:25:34.868 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/connection.py", line 940, in _mexe 2020-11-09T18:25:34.869 INFO:tasks.rgw_multisite_tests: request.body, request.headers) 2020-11-09T18:25:34.869 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 884, in sender 2020-11-09T18:25:34.869 INFO:tasks.rgw_multisite_tests: response.status, response.reason, body) 2020-11-09T18:25:34.869 INFO:tasks.rgw_multisite_tests:boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request 2020-11-09T18:25:34.870 INFO:tasks.rgw_multisite_tests:InvalidArgumentymzeui-49tx000000000000000000007-005fa987a1-122c-test-zone1122c-test-zone1-test-zonegroup 2020-11-09T18:25:34.870 INFO:tasks.rgw_multisite_tests: 2020-11-09T18:25:34.870 INFO:tasks.rgw_multisite_tests:====================================================================== 2020-11-09T18:25:34.870 INFO:tasks.rgw_multisite_tests:ERROR: tasks.rgw_multi.tests.test_bucket_sync_enable_right_after_disable 2020-11-09T18:25:34.871 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-11-09T18:25:34.871 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2020-11-09T18:25:34.871 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2020-11-09T18:25:34.871 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2020-11-09T18:25:34.872 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-dang-webdir-empty-object/qa/tasks/rgw_multi/tests.py", line 1127, in test_bucket_sync_enable_right_after_disable 2020-11-09T18:25:34.872 INFO:tasks.rgw_multisite_tests: k.set_contents_from_string(content) 2020-11-09T18:25:34.872 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 1426, in set_contents_from_string 2020-11-09T18:25:34.872 INFO:tasks.rgw_multisite_tests: encrypt_key=encrypt_key) 2020-11-09T18:25:34.873 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file 2020-11-09T18:25:34.873 INFO:tasks.rgw_multisite_tests: chunked_transfer=chunked_transfer, size=size) 2020-11-09T18:25:34.873 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 750, in send_file 2020-11-09T18:25:34.873 INFO:tasks.rgw_multisite_tests: chunked_transfer=chunked_transfer, size=size) 2020-11-09T18:25:34.873 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 951, in _send_file_internal 2020-11-09T18:25:34.874 INFO:tasks.rgw_multisite_tests: query_args=query_args 2020-11-09T18:25:34.874 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/connection.py", line 668, in make_request 2020-11-09T18:25:34.874 INFO:tasks.rgw_multisite_tests: retry_handler=retry_handler 2020-11-09T18:25:34.874 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/connection.py", line 1071, in make_request 2020-11-09T18:25:34.875 INFO:tasks.rgw_multisite_tests: retry_handler=retry_handler) 2020-11-09T18:25:34.875 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/connection.py", line 940, in _mexe 2020-11-09T18:25:34.875 INFO:tasks.rgw_multisite_tests: request.body, request.headers) 2020-11-09T18:25:34.875 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 884, in sender 2020-11-09T18:25:34.876 INFO:tasks.rgw_multisite_tests: response.status, response.reason, body) 2020-11-09T18:25:34.876 INFO:tasks.rgw_multisite_tests:boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request 2020-11-09T18:25:34.876 INFO:tasks.rgw_multisite_tests:InvalidArgumentymzeui-59tx00000000000000000041a-005fa9899e-128a-test-zone1128a-test-zone1-test-zonegroup 2020-11-09T18:25:34.876 INFO:tasks.rgw_multisite_tests: 2020-11-09T18:25:34.877 INFO:tasks.rgw_multisite_tests:====================================================================== 2020-11-09T18:25:34.877 INFO:tasks.rgw_multisite_tests:ERROR: tasks.rgw_multi.tests.test_bucket_sync_disable_enable 2020-11-09T18:25:34.877 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-11-09T18:25:34.877 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2020-11-09T18:25:34.877 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2020-11-09T18:25:34.878 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2020-11-09T18:25:34.878 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-dang-webdir-empty-object/qa/tasks/rgw_multi/tests.py", line 1161, in test_bucket_sync_disable_enable 2020-11-09T18:25:34.878 INFO:tasks.rgw_multisite_tests: k.set_contents_from_string(content) 2020-11-09T18:25:34.878 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 1426, in set_contents_from_string 2020-11-09T18:25:34.879 INFO:tasks.rgw_multisite_tests: encrypt_key=encrypt_key) 2020-11-09T18:25:34.879 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file 2020-11-09T18:25:34.879 INFO:tasks.rgw_multisite_tests: chunked_transfer=chunked_transfer, size=size) 2020-11-09T18:25:34.879 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 750, in send_file 2020-11-09T18:25:34.880 INFO:tasks.rgw_multisite_tests: chunked_transfer=chunked_transfer, size=size) 2020-11-09T18:25:34.880 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 951, in _send_file_internal 2020-11-09T18:25:34.880 INFO:tasks.rgw_multisite_tests: query_args=query_args 2020-11-09T18:25:34.880 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/connection.py", line 668, in make_request 2020-11-09T18:25:34.880 INFO:tasks.rgw_multisite_tests: retry_handler=retry_handler 2020-11-09T18:25:34.881 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/connection.py", line 1071, in make_request 2020-11-09T18:25:34.881 INFO:tasks.rgw_multisite_tests: retry_handler=retry_handler) 2020-11-09T18:25:34.881 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/connection.py", line 940, in _mexe 2020-11-09T18:25:34.881 INFO:tasks.rgw_multisite_tests: request.body, request.headers) 2020-11-09T18:25:34.882 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 884, in sender 2020-11-09T18:25:34.882 INFO:tasks.rgw_multisite_tests: response.status, response.reason, body) 2020-11-09T18:25:34.882 INFO:tasks.rgw_multisite_tests:boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request 2020-11-09T18:25:34.882 INFO:tasks.rgw_multisite_tests:InvalidArgumentymzeui-62tx000000000000000000447-005fa9899e-128a-test-zone1128a-test-zone1-test-zonegroup 2020-11-09T18:25:34.883 INFO:tasks.rgw_multisite_tests: 2020-11-09T18:25:34.883 INFO:tasks.rgw_multisite_tests:====================================================================== 2020-11-09T18:25:34.883 INFO:tasks.rgw_multisite_tests:ERROR: tasks.rgw_multi.tests.test_multipart_object_sync 2020-11-09T18:25:34.883 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-11-09T18:25:34.884 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2020-11-09T18:25:34.884 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2020-11-09T18:25:34.884 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2020-11-09T18:25:34.884 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-dang-webdir-empty-object/qa/tasks/rgw_multi/tests.py", line 1196, in test_multipart_object_sync 2020-11-09T18:25:34.885 INFO:tasks.rgw_multisite_tests: upload = bucket.initiate_multipart_upload('MULTIPART') 2020-11-09T18:25:34.885 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/bucket.py", line 1766, in initiate_multipart_upload 2020-11-09T18:25:34.885 INFO:tasks.rgw_multisite_tests: response.status, response.reason, body) 2020-11-09T18:25:34.885 INFO:tasks.rgw_multisite_tests:boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request 2020-11-09T18:25:34.885 INFO:tasks.rgw_multisite_tests:InvalidArgumentRequest Content-Type is not multipart/form-dataymzeui-65tx00000000000000000045a-005fa9899e-128a-test-zone1128a-test-zone1-test-zonegroup 2020-11-09T18:25:34.886 INFO:tasks.rgw_multisite_tests: 2020-11-09T18:25:34.886 INFO:tasks.rgw_multisite_tests:====================================================================== 2020-11-09T18:25:34.886 INFO:tasks.rgw_multisite_tests:ERROR: tasks.rgw_multi.tests.test_encrypted_object_sync 2020-11-09T18:25:34.886 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-11-09T18:25:34.887 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2020-11-09T18:25:34.887 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2020-11-09T18:25:34.887 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2020-11-09T18:25:34.887 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-dang-webdir-empty-object/qa/tasks/rgw_multi/tests.py", line 1231, in test_encrypted_object_sync 2020-11-09T18:25:34.888 INFO:tasks.rgw_multisite_tests: key.set_contents_from_string(data, headers=sse_c_headers) 2020-11-09T18:25:34.888 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 1426, in set_contents_from_string 2020-11-09T18:25:34.888 INFO:tasks.rgw_multisite_tests: encrypt_key=encrypt_key) 2020-11-09T18:25:34.888 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file 2020-11-09T18:25:34.889 INFO:tasks.rgw_multisite_tests: chunked_transfer=chunked_transfer, size=size) 2020-11-09T18:25:34.889 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 750, in send_file 2020-11-09T18:25:34.889 INFO:tasks.rgw_multisite_tests: chunked_transfer=chunked_transfer, size=size) 2020-11-09T18:25:34.889 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 951, in _send_file_internal 2020-11-09T18:25:34.889 INFO:tasks.rgw_multisite_tests: query_args=query_args 2020-11-09T18:25:34.890 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/connection.py", line 668, in make_request 2020-11-09T18:25:34.890 INFO:tasks.rgw_multisite_tests: retry_handler=retry_handler 2020-11-09T18:25:34.890 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/connection.py", line 1071, in make_request 2020-11-09T18:25:34.890 INFO:tasks.rgw_multisite_tests: retry_handler=retry_handler) 2020-11-09T18:25:34.891 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/connection.py", line 940, in _mexe 2020-11-09T18:25:34.891 INFO:tasks.rgw_multisite_tests: request.body, request.headers) 2020-11-09T18:25:34.891 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 884, in sender 2020-11-09T18:25:34.891 INFO:tasks.rgw_multisite_tests: response.status, response.reason, body) 2020-11-09T18:25:34.892 INFO:tasks.rgw_multisite_tests:boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request 2020-11-09T18:25:34.892 INFO:tasks.rgw_multisite_tests:InvalidArgumentymzeui-68tx00000000000000000045c-005fa9899e-128a-test-zone1128a-test-zone1-test-zonegroup 2020-11-09T18:25:34.892 INFO:tasks.rgw_multisite_tests: 2020-11-09T18:25:34.892 INFO:tasks.rgw_multisite_tests:====================================================================== 2020-11-09T18:25:34.892 INFO:tasks.rgw_multisite_tests:ERROR: tasks.rgw_multi.tests.test_bucket_index_log_trim 2020-11-09T18:25:34.893 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-11-09T18:25:34.893 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2020-11-09T18:25:34.893 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2020-11-09T18:25:34.893 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2020-11-09T18:25:34.894 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-dang-webdir-empty-object/qa/tasks/rgw_multi/tests.py", line 1273, in test_bucket_index_log_trim 2020-11-09T18:25:34.894 INFO:tasks.rgw_multisite_tests: cold_bucket = make_test_bucket() 2020-11-09T18:25:34.894 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-dang-webdir-empty-object/qa/tasks/rgw_multi/tests.py", line 1267, in make_test_bucket 2020-11-09T18:25:34.894 INFO:tasks.rgw_multisite_tests: k.set_contents_from_string('foo') 2020-11-09T18:25:34.895 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 1426, in set_contents_from_string 2020-11-09T18:25:34.895 INFO:tasks.rgw_multisite_tests: encrypt_key=encrypt_key) 2020-11-09T18:25:34.895 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file 2020-11-09T18:25:34.895 INFO:tasks.rgw_multisite_tests: chunked_transfer=chunked_transfer, size=size) 2020-11-09T18:25:34.896 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 750, in send_file 2020-11-09T18:25:34.896 INFO:tasks.rgw_multisite_tests: chunked_transfer=chunked_transfer, size=size) 2020-11-09T18:25:34.896 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 951, in _send_file_internal 2020-11-09T18:25:34.896 INFO:tasks.rgw_multisite_tests: query_args=query_args 2020-11-09T18:25:34.896 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/connection.py", line 668, in make_request 2020-11-09T18:25:34.897 INFO:tasks.rgw_multisite_tests: retry_handler=retry_handler 2020-11-09T18:25:34.897 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/connection.py", line 1071, in make_request 2020-11-09T18:25:34.897 INFO:tasks.rgw_multisite_tests: retry_handler=retry_handler) 2020-11-09T18:25:34.897 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/connection.py", line 940, in _mexe 2020-11-09T18:25:34.898 INFO:tasks.rgw_multisite_tests: request.body, request.headers) 2020-11-09T18:25:34.898 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/boto/s3/key.py", line 884, in sender 2020-11-09T18:25:34.898 INFO:tasks.rgw_multisite_tests: response.status, response.reason, body) 2020-11-09T18:25:34.898 INFO:tasks.rgw_multisite_tests:boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request 2020-11-09T18:25:34.898 INFO:tasks.rgw_multisite_tests:InvalidArgumentymzeui-69tx00000000000000000046e-005fa9899e-128a-test-zone1128a-test-zone1-test-zonegroup 2020-11-09T18:25:34.899 INFO:tasks.rgw_multisite_tests: 2020-11-09T18:25:34.899 INFO:tasks.rgw_multisite_tests:====================================================================== 2020-11-09T18:25:34.899 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_bucket_sync_disable 2020-11-09T18:25:34.899 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-11-09T18:25:34.900 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2020-11-09T18:25:34.900 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2020-11-09T18:25:34.900 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2020-11-09T18:25:34.900 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-dang-webdir-empty-object/qa/tasks/rgw_multi/tests.py", line 1114, in test_bucket_sync_disable 2020-11-09T18:25:34.901 INFO:tasks.rgw_multisite_tests: zonegroup_data_checkpoint(zonegroup_conns) 2020-11-09T18:25:34.901 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-dang-webdir-empty-object/qa/tasks/rgw_multi/tests.py", line 367, in zonegroup_data_checkpoint 2020-11-09T18:25:34.901 INFO:tasks.rgw_multisite_tests: zone_data_checkpoint(target_conn.zone, source_conn.zone) 2020-11-09T18:25:34.901 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-dang-webdir-empty-object/qa/tasks/rgw_multi/tests.py", line 359, in zone_data_checkpoint 2020-11-09T18:25:34.902 INFO:tasks.rgw_multisite_tests: (target_zone.name, source_zone.name) 2020-11-09T18:25:34.902 INFO:tasks.rgw_multisite_tests:AssertionError: failed data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-11-09T18:25:34.902 INFO:tasks.rgw_multisite_tests: 2020-11-09T18:25:34.902 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-11-09T18:25:34.903 INFO:tasks.rgw_multisite_tests:Ran 28 tests in 775.241s 2020-11-09T18:25:34.903 INFO:tasks.rgw_multisite_tests: 2020-11-09T18:25:34.903 INFO:tasks.rgw_multisite_tests:FAILED (errors=15, failures=1) 2020-11-09T18:25:34.906 INFO:tasks.rgw_multisite_tests:log information for manual testing ... ok 2020-11-09T18:25:34.907 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:34.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:34.908 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:25:34.949 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:25:34.949 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:25:35.160 INFO:teuthology.orchestra.run.smithi051.stderr:No --period given, using current period=f6258aba-300a-4d5d-ad93-a7c198a0e0a1 2020-11-09T18:25:35.194 INFO:teuthology.orchestra.run.smithi051.stdout:[ 2020-11-09T18:25:35.194 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:25:35.194 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604946334.696681_428.1", 2020-11-09T18:25:35.195 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "2020-11-09T18:25:34.696681Z" 2020-11-09T18:25:35.195 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:25:35.195 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:25:35.195 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:25:35.195 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:25:35.195 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:25:35.196 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:25:35.196 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:25:35.196 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:25:35.196 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:25:35.196 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:25:35.196 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:25:35.196 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:25:35.197 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:25:35.197 INFO:teuthology.orchestra.run.smithi051.stdout:] 2020-11-09T18:25:35.198 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-09T18:25:35.198 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:35.199 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:35.199 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:25:35.239 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:25:35.240 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:25:35.512 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:25:35.513 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:25:35.513 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:25:35.513 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:25:35.513 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:25:35.513 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:25:35.514 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:25:35.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:35.514 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:25:35.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:35.514 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:25:35.514 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:35.514 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:25:35.515 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946334.696681_428.1", 2020-11-09T18:25:35.515 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:35.515 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:25:35.515 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:35.515 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:34.696681Z", 2020-11-09T18:25:35.515 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:25:35.515 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:35.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:35.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:35.516 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:25:35.516 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:35.516 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:25:35.516 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:25:35.516 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:35.517 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:35.517 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:35.517 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:25:35.517 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:25:35.517 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:35.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:35.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:35.518 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:25:35.518 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:35.518 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:25:35.518 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:25:35.518 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:35.518 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:35.518 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:35.519 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:25:35.519 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:25:35.519 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:35.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:35.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:35.519 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:25:35.519 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:35.520 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:25:35.520 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:25:35.520 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:35.520 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:35.520 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:35.520 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:25:35.520 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:25:35.521 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:35.521 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:35.521 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:25:35.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:35.521 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:25:35.521 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:25:35.521 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:25:35.521 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:35.522 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:25:35.523 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-09T18:25:35.523 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-09T18:25:35.524 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:35.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:35.525 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:25:35.571 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:25:35.571 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:25:35.821 INFO:teuthology.orchestra.run.smithi051.stdout:{ 2020-11-09T18:25:35.821 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_status": { 2020-11-09T18:25:35.822 INFO:teuthology.orchestra.run.smithi051.stdout: "info": { 2020-11-09T18:25:35.822 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "sync", 2020-11-09T18:25:35.822 INFO:teuthology.orchestra.run.smithi051.stdout: "num_shards": 4, 2020-11-09T18:25:35.822 INFO:teuthology.orchestra.run.smithi051.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:25:35.822 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 4 2020-11-09T18:25:35.822 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:25:35.823 INFO:teuthology.orchestra.run.smithi051.stdout: "markers": [ 2020-11-09T18:25:35.823 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:25:35.823 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 0, 2020-11-09T18:25:35.823 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:25:35.823 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:25:35.823 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604946334.696681_428.1", 2020-11-09T18:25:35.823 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:25:35.824 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 1, 2020-11-09T18:25:35.824 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:25:35.824 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:25:34.696681Z", 2020-11-09T18:25:35.824 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 4 2020-11-09T18:25:35.824 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:25:35.824 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:25:35.824 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:25:35.825 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 1, 2020-11-09T18:25:35.825 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:25:35.825 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:25:35.825 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:25:35.825 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:25:35.825 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:25:35.825 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:25:35.825 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:25:35.826 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:25:35.826 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:25:35.826 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:25:35.826 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:25:35.826 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 2, 2020-11-09T18:25:35.826 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:25:35.826 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:25:35.827 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:25:35.827 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:25:35.827 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:25:35.827 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:25:35.827 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:25:35.827 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:25:35.827 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:25:35.828 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:25:35.828 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:25:35.828 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 3, 2020-11-09T18:25:35.828 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:25:35.828 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:25:35.828 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:25:35.828 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:25:35.829 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:25:35.829 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:25:35.829 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:25:35.829 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:25:35.829 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:25:35.829 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:25:35.829 INFO:teuthology.orchestra.run.smithi051.stdout: ] 2020-11-09T18:25:35.830 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:25:35.830 INFO:teuthology.orchestra.run.smithi051.stdout: "full_sync": { 2020-11-09T18:25:35.830 INFO:teuthology.orchestra.run.smithi051.stdout: "total": 1, 2020-11-09T18:25:35.830 INFO:teuthology.orchestra.run.smithi051.stdout: "complete": 1 2020-11-09T18:25:35.830 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:25:35.830 INFO:teuthology.orchestra.run.smithi051.stdout:} 2020-11-09T18:25:35.831 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-09T18:25:35.832 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-09T18:25:35.832 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:35.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:35.833 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:25:35.872 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:25:35.873 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:25:36.239 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:25:36.240 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:25:36.240 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:25:36.241 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:25:36.241 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:25:36.241 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:25:36.241 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:25:36.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:36.242 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:25:36.242 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:36.242 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:25:36.242 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:36.242 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:25:36.243 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946334.696681_428.1", 2020-11-09T18:25:36.243 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:36.243 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:25:36.243 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:36.243 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:34.696681Z", 2020-11-09T18:25:36.243 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:25:36.244 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:36.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:36.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:36.244 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:25:36.244 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:36.244 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:25:36.244 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:25:36.245 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:36.245 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:36.245 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:36.245 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:25:36.245 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:25:36.245 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:36.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:36.246 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:36.246 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:25:36.246 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:36.246 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:25:36.246 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:25:36.246 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:36.247 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:36.247 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:36.247 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:25:36.247 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:25:36.247 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:36.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:36.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:36.248 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:25:36.248 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:36.248 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:25:36.248 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:25:36.248 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:36.249 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:36.249 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:36.249 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:25:36.249 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:25:36.249 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:36.250 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:36.250 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:25:36.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:36.250 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:25:36.250 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:25:36.250 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:25:36.250 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:36.251 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:25:36.252 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-09T18:25:36.252 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:36.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:36.253 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:25:36.296 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:25:36.296 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:25:36.530 INFO:teuthology.orchestra.run.smithi051.stderr:No --period given, using current period=f6258aba-300a-4d5d-ad93-a7c198a0e0a1 2020-11-09T18:25:36.562 INFO:teuthology.orchestra.run.smithi051.stdout:[ 2020-11-09T18:25:36.562 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:25:36.562 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604946334.696681_428.1", 2020-11-09T18:25:36.562 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "2020-11-09T18:25:34.696681Z" 2020-11-09T18:25:36.563 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:25:36.563 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:25:36.563 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:25:36.563 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:25:36.563 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:25:36.563 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:25:36.564 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:25:36.564 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:25:36.564 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:25:36.564 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:25:36.564 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:25:36.564 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:25:36.564 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:25:36.564 INFO:teuthology.orchestra.run.smithi051.stdout:] 2020-11-09T18:25:36.565 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-09T18:25:36.566 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:36.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:36.567 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:25:36.607 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:25:36.607 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:25:36.871 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:25:36.872 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:25:36.872 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:25:36.872 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:25:36.873 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:25:36.873 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:25:36.873 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:25:36.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:36.873 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:25:36.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:36.874 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:25:36.874 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:36.874 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:25:36.874 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946334.696681_428.1", 2020-11-09T18:25:36.875 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:36.875 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:25:36.875 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:36.875 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:34.696681Z", 2020-11-09T18:25:36.875 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:25:36.875 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:36.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:36.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:36.876 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:25:36.876 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:36.876 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:25:36.876 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:25:36.876 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:36.876 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:36.877 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:36.877 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:25:36.877 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:25:36.877 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:36.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:36.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:36.877 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:25:36.878 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:36.878 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:25:36.878 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:25:36.878 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:36.878 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:36.878 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:36.878 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:25:36.879 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:25:36.879 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:36.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:36.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:36.879 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:25:36.879 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:36.879 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:25:36.880 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:25:36.880 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:36.880 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:36.880 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:36.880 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:25:36.880 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:25:36.880 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:36.881 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:36.881 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:25:36.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:36.881 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:25:36.882 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:25:36.882 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:25:36.882 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:36.882 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:25:36.883 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-09T18:25:37.743 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:37.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:37.744 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:25:37.778 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:25:37.779 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:25:37.981 INFO:teuthology.orchestra.run.smithi051.stderr:No --period given, using current period=f6258aba-300a-4d5d-ad93-a7c198a0e0a1 2020-11-09T18:25:38.016 INFO:teuthology.orchestra.run.smithi051.stdout:[ 2020-11-09T18:25:38.016 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:25:38.016 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:25:38.017 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "2020-11-09T18:25:37.732873Z" 2020-11-09T18:25:38.017 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:25:38.017 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:25:38.017 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:25:38.017 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:25:38.017 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:25:38.017 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:25:38.018 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:25:38.018 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:25:38.018 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:25:38.018 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:25:38.018 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:25:38.018 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:25:38.018 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:25:38.019 INFO:teuthology.orchestra.run.smithi051.stdout:] 2020-11-09T18:25:38.019 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-09T18:25:38.020 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:38.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:38.021 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:25:38.062 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:25:38.063 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:25:38.327 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:25:38.328 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:25:38.328 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:25:38.328 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:25:38.328 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:25:38.329 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:25:38.329 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:25:38.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:38.329 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:25:38.329 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:38.329 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:25:38.329 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:38.330 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:25:38.330 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:25:38.330 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:38.330 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:25:38.331 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:38.331 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:25:38.331 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:25:38.332 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:38.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:38.332 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:38.332 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:25:38.332 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:38.332 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:25:38.332 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:25:38.333 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:38.333 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:38.333 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:38.333 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:25:38.333 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:25:38.333 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:38.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:38.334 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:38.334 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:25:38.334 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:38.334 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:25:38.334 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:25:38.334 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:38.334 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:38.335 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:38.335 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:25:38.335 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:25:38.335 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:38.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:38.335 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:38.335 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:25:38.336 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:38.336 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:25:38.336 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:25:38.336 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:38.336 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:38.336 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:38.336 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:25:38.337 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:25:38.337 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:38.337 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:38.337 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:25:38.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:38.337 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:25:38.337 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:25:38.337 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:25:38.338 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:38.338 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:25:38.339 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-09T18:25:38.341 INFO:tasks.rgw.c2.client.1.smithi078.stdout:*** Caught signal (Segmentation fault) ** 2020-11-09T18:25:38.341 INFO:tasks.rgw.c2.client.1.smithi078.stdout: in thread 7f26b67fc700 thread_name:civetweb-worker 2020-11-09T18:25:38.344 INFO:tasks.rgw.c2.client.1.smithi078.stdout: ceph version 16.0.0-7048-ga0560c8e (a0560c8e5c6c33c4655e9d2a8b022793ae57419f) pacific (dev) 2020-11-09T18:25:38.344 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7f27d8d23980] 2020-11-09T18:25:38.344 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 2: (RGWHandler_REST_PSTopic::op_put()+0xb) [0x7f27d96b468b] 2020-11-09T18:25:38.345 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 3: (RGWHandler_REST::get_op()+0x96) [0x7f27d97b6f56] 2020-11-09T18:25:38.345 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 4: (process_request(rgw::sal::RGWRadosStore*, RGWREST*, RGWRequest*, std::__cxx11::basic_string, std::allocator > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, int*)+0x5c1) [0x7f27d93b96e1] 2020-11-09T18:25:38.345 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 5: (RGWCivetWebFrontend::process(mg_connection*)+0x278) [0x7f27d92e7658] 2020-11-09T18:25:38.345 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 6: /usr/lib/libradosgw.so.2(+0x550dce) [0x7f27d9480dce] 2020-11-09T18:25:38.345 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 7: /usr/lib/libradosgw.so.2(+0x552a67) [0x7f27d9482a67] 2020-11-09T18:25:38.346 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 8: /usr/lib/libradosgw.so.2(+0x552f08) [0x7f27d9482f08] 2020-11-09T18:25:38.346 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 9: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f27d8d186db] 2020-11-09T18:25:38.346 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 10: clone() 2020-11-09T18:25:38.346 INFO:tasks.rgw.c2.client.1.smithi078.stdout:2020-11-09T18:25:38.344+0000 7f26b67fc700 -1 *** Caught signal (Segmentation fault) ** 2020-11-09T18:25:38.347 INFO:tasks.rgw.c2.client.1.smithi078.stdout: in thread 7f26b67fc700 thread_name:civetweb-worker 2020-11-09T18:25:38.347 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 2020-11-09T18:25:38.347 INFO:tasks.rgw.c2.client.1.smithi078.stdout: ceph version 16.0.0-7048-ga0560c8e (a0560c8e5c6c33c4655e9d2a8b022793ae57419f) pacific (dev) 2020-11-09T18:25:38.347 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7f27d8d23980] 2020-11-09T18:25:38.347 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 2: (RGWHandler_REST_PSTopic::op_put()+0xb) [0x7f27d96b468b] 2020-11-09T18:25:38.347 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 3: (RGWHandler_REST::get_op()+0x96) [0x7f27d97b6f56] 2020-11-09T18:25:38.348 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 4: (process_request(rgw::sal::RGWRadosStore*, RGWREST*, RGWRequest*, std::__cxx11::basic_string, std::allocator > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, int*)+0x5c1) [0x7f27d93b96e1] 2020-11-09T18:25:38.348 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 5: (RGWCivetWebFrontend::process(mg_connection*)+0x278) [0x7f27d92e7658] 2020-11-09T18:25:38.348 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 6: /usr/lib/libradosgw.so.2(+0x550dce) [0x7f27d9480dce] 2020-11-09T18:25:38.348 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 7: /usr/lib/libradosgw.so.2(+0x552a67) [0x7f27d9482a67] 2020-11-09T18:25:38.348 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 8: /usr/lib/libradosgw.so.2(+0x552f08) [0x7f27d9482f08] 2020-11-09T18:25:38.348 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 9: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f27d8d186db] 2020-11-09T18:25:38.349 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 10: clone() 2020-11-09T18:25:38.349 INFO:tasks.rgw.c2.client.1.smithi078.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2020-11-09T18:25:38.349 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 2020-11-09T18:25:38.364 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 0> 2020-11-09T18:25:38.344+0000 7f26b67fc700 -1 *** Caught signal (Segmentation fault) ** 2020-11-09T18:25:38.364 INFO:tasks.rgw.c2.client.1.smithi078.stdout: in thread 7f26b67fc700 thread_name:civetweb-worker 2020-11-09T18:25:38.365 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 2020-11-09T18:25:38.365 INFO:tasks.rgw.c2.client.1.smithi078.stdout: ceph version 16.0.0-7048-ga0560c8e (a0560c8e5c6c33c4655e9d2a8b022793ae57419f) pacific (dev) 2020-11-09T18:25:38.365 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7f27d8d23980] 2020-11-09T18:25:38.365 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 2: (RGWHandler_REST_PSTopic::op_put()+0xb) [0x7f27d96b468b] 2020-11-09T18:25:38.365 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 3: (RGWHandler_REST::get_op()+0x96) [0x7f27d97b6f56] 2020-11-09T18:25:38.365 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 4: (process_request(rgw::sal::RGWRadosStore*, RGWREST*, RGWRequest*, std::__cxx11::basic_string, std::allocator > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, int*)+0x5c1) [0x7f27d93b96e1] 2020-11-09T18:25:38.366 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 5: (RGWCivetWebFrontend::process(mg_connection*)+0x278) [0x7f27d92e7658] 2020-11-09T18:25:38.366 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 6: /usr/lib/libradosgw.so.2(+0x550dce) [0x7f27d9480dce] 2020-11-09T18:25:38.366 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 7: /usr/lib/libradosgw.so.2(+0x552a67) [0x7f27d9482a67] 2020-11-09T18:25:38.366 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 8: /usr/lib/libradosgw.so.2(+0x552f08) [0x7f27d9482f08] 2020-11-09T18:25:38.366 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 9: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f27d8d186db] 2020-11-09T18:25:38.366 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 10: clone() 2020-11-09T18:25:38.366 INFO:tasks.rgw.c2.client.1.smithi078.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2020-11-09T18:25:38.367 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 2020-11-09T18:25:38.380 INFO:tasks.rgw.c2.client.1.smithi078.stdout: -1> 2020-11-09T18:25:38.344+0000 7f26b67fc700 -1 *** Caught signal (Segmentation fault) ** 2020-11-09T18:25:38.381 INFO:tasks.rgw.c2.client.1.smithi078.stdout: in thread 7f26b67fc700 thread_name:civetweb-worker 2020-11-09T18:25:38.381 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 2020-11-09T18:25:38.381 INFO:tasks.rgw.c2.client.1.smithi078.stdout: ceph version 16.0.0-7048-ga0560c8e (a0560c8e5c6c33c4655e9d2a8b022793ae57419f) pacific (dev) 2020-11-09T18:25:38.381 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7f27d8d23980] 2020-11-09T18:25:38.381 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 2: (RGWHandler_REST_PSTopic::op_put()+0xb) [0x7f27d96b468b] 2020-11-09T18:25:38.381 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 3: (RGWHandler_REST::get_op()+0x96) [0x7f27d97b6f56] 2020-11-09T18:25:38.381 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 4: (process_request(rgw::sal::RGWRadosStore*, RGWREST*, RGWRequest*, std::__cxx11::basic_string, std::allocator > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, int*)+0x5c1) [0x7f27d93b96e1] 2020-11-09T18:25:38.382 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 5: (RGWCivetWebFrontend::process(mg_connection*)+0x278) [0x7f27d92e7658] 2020-11-09T18:25:38.382 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 6: /usr/lib/libradosgw.so.2(+0x550dce) [0x7f27d9480dce] 2020-11-09T18:25:38.382 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 7: /usr/lib/libradosgw.so.2(+0x552a67) [0x7f27d9482a67] 2020-11-09T18:25:38.382 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 8: /usr/lib/libradosgw.so.2(+0x552f08) [0x7f27d9482f08] 2020-11-09T18:25:38.382 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 9: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f27d8d186db] 2020-11-09T18:25:38.382 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 10: clone() 2020-11-09T18:25:38.382 INFO:tasks.rgw.c2.client.1.smithi078.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2020-11-09T18:25:38.383 INFO:tasks.rgw.c2.client.1.smithi078.stdout: 2020-11-09T18:25:38.395 INFO:tasks.rgw_multisite_tests:test low level implementation of s3 notifications ... ERROR 2020-11-09T18:25:38.396 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:38.396 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:38.397 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:25:38.434 INFO:tasks.rgw.c2.client.1.smithi078.stderr:daemon-helper: command crashed with signal 11 2020-11-09T18:25:38.438 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:25:38.438 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:25:38.662 INFO:teuthology.orchestra.run.smithi051.stderr:No --period given, using current period=f6258aba-300a-4d5d-ad93-a7c198a0e0a1 2020-11-09T18:25:38.698 INFO:teuthology.orchestra.run.smithi051.stdout:[ 2020-11-09T18:25:38.698 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:25:38.699 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:25:38.700 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "2020-11-09T18:25:37.732873Z" 2020-11-09T18:25:38.700 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:25:38.700 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:25:38.700 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:25:38.700 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:25:38.700 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:25:38.701 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:25:38.701 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:25:38.701 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:25:38.701 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:25:38.701 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:25:38.701 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:25:38.701 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:25:38.702 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:25:38.702 INFO:teuthology.orchestra.run.smithi051.stdout:] 2020-11-09T18:25:38.703 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-09T18:25:38.703 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:38.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:38.704 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:25:38.744 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:25:38.745 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:25:39.006 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:25:39.006 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:25:39.006 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:25:39.007 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:25:39.007 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:25:39.007 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:25:39.007 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:25:39.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:39.008 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:25:39.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:39.008 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:25:39.008 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:39.008 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:25:39.009 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:25:39.009 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:39.009 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:25:39.009 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:39.009 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:25:39.010 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:25:39.010 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:39.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:39.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:39.010 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:25:39.011 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:39.011 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:25:39.011 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:25:39.011 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:39.011 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:39.011 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:39.011 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:25:39.012 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:25:39.012 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:39.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:39.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:39.012 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:25:39.012 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:39.012 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:25:39.013 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:25:39.013 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:39.013 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:39.013 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:39.013 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:25:39.013 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:25:39.013 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:39.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:39.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:39.014 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:25:39.014 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:39.014 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:25:39.014 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:25:39.014 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:39.015 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:39.015 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:39.015 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:25:39.015 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:25:39.015 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:39.015 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:39.016 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:25:39.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:39.016 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:25:39.016 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:25:39.016 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:25:39.016 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:39.016 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:25:39.017 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-09T18:25:39.018 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-09T18:25:39.018 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:39.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:39.019 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:25:39.059 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:25:39.059 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:25:39.339 INFO:teuthology.orchestra.run.smithi051.stdout:{ 2020-11-09T18:25:39.340 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_status": { 2020-11-09T18:25:39.340 INFO:teuthology.orchestra.run.smithi051.stdout: "info": { 2020-11-09T18:25:39.340 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "sync", 2020-11-09T18:25:39.340 INFO:teuthology.orchestra.run.smithi051.stdout: "num_shards": 4, 2020-11-09T18:25:39.340 INFO:teuthology.orchestra.run.smithi051.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:25:39.340 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 4 2020-11-09T18:25:39.341 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:25:39.341 INFO:teuthology.orchestra.run.smithi051.stdout: "markers": [ 2020-11-09T18:25:39.341 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:25:39.341 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 0, 2020-11-09T18:25:39.341 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:25:39.341 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:25:39.341 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:25:39.342 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:25:39.342 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 1, 2020-11-09T18:25:39.342 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:25:39.342 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:25:39.342 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 4 2020-11-09T18:25:39.342 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:25:39.342 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:25:39.343 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:25:39.343 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 1, 2020-11-09T18:25:39.343 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:25:39.343 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:25:39.343 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:25:39.343 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:25:39.343 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:25:39.344 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:25:39.344 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:25:39.344 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:25:39.344 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:25:39.344 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:25:39.344 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:25:39.344 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 2, 2020-11-09T18:25:39.344 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:25:39.345 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:25:39.345 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:25:39.345 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:25:39.345 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:25:39.345 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:25:39.345 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:25:39.345 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:25:39.346 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:25:39.346 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:25:39.346 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:25:39.346 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 3, 2020-11-09T18:25:39.346 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:25:39.346 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:25:39.346 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:25:39.347 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:25:39.347 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:25:39.347 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:25:39.347 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:25:39.347 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:25:39.347 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:25:39.347 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:25:39.348 INFO:teuthology.orchestra.run.smithi051.stdout: ] 2020-11-09T18:25:39.348 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:25:39.348 INFO:teuthology.orchestra.run.smithi051.stdout: "full_sync": { 2020-11-09T18:25:39.348 INFO:teuthology.orchestra.run.smithi051.stdout: "total": 1, 2020-11-09T18:25:39.348 INFO:teuthology.orchestra.run.smithi051.stdout: "complete": 1 2020-11-09T18:25:39.348 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:25:39.348 INFO:teuthology.orchestra.run.smithi051.stdout:} 2020-11-09T18:25:39.349 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-09T18:25:39.350 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-09T18:25:39.350 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:39.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:39.351 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:25:39.392 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:25:39.393 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:25:39.664 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:25:39.664 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:25:39.664 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:25:39.664 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:25:39.665 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:25:39.665 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:25:39.665 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:25:39.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:39.665 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:25:39.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:39.665 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:25:39.666 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:39.666 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:25:39.666 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:25:39.666 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:39.666 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:25:39.666 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:39.666 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:25:39.667 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:25:39.667 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:39.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:39.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:39.667 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:25:39.667 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:39.667 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:25:39.668 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:25:39.668 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:39.668 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:39.668 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:39.668 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:25:39.668 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:25:39.668 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:39.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:39.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:39.669 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:25:39.669 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:39.669 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:25:39.669 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:25:39.669 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:39.670 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:39.670 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:39.670 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:25:39.670 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:25:39.670 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:39.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:39.670 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:39.670 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:25:39.671 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:39.671 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:25:39.671 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:25:39.671 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:39.671 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:39.671 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:39.671 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:25:39.672 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:25:39.672 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:39.672 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:39.672 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:25:39.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:39.672 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:25:39.672 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:25:39.673 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:25:39.673 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:39.673 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:25:39.674 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-09T18:25:39.675 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:39.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:39.676 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:25:39.715 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:25:39.715 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:25:39.933 INFO:teuthology.orchestra.run.smithi051.stderr:No --period given, using current period=f6258aba-300a-4d5d-ad93-a7c198a0e0a1 2020-11-09T18:25:39.967 INFO:teuthology.orchestra.run.smithi051.stdout:[ 2020-11-09T18:25:39.967 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:25:39.968 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:25:39.968 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "2020-11-09T18:25:37.732873Z" 2020-11-09T18:25:39.968 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:25:39.968 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:25:39.968 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:25:39.968 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:25:39.968 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:25:39.969 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:25:39.969 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:25:39.969 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:25:39.969 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:25:39.969 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:25:39.969 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:25:39.969 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:25:39.970 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:25:39.970 INFO:teuthology.orchestra.run.smithi051.stdout:] 2020-11-09T18:25:39.971 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-09T18:25:39.971 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:39.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:39.972 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:25:40.013 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:25:40.013 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:25:40.288 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:25:40.288 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:25:40.289 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:25:40.289 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:25:40.289 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:25:40.289 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:25:40.289 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:25:40.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:40.290 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:25:40.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:40.290 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:25:40.290 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:40.290 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:25:40.290 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:25:40.290 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:40.291 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:25:40.291 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:40.291 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:25:40.291 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:25:40.291 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:40.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:40.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:40.292 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:25:40.292 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:40.292 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:25:40.292 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:25:40.292 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:40.292 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:40.292 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:40.292 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:25:40.293 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:25:40.293 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:40.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:40.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:40.293 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:25:40.293 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:40.293 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:25:40.294 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:25:40.294 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:40.294 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:40.294 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:40.294 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:25:40.294 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:25:40.294 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:40.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:40.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:40.295 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:25:40.295 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:40.295 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:25:40.295 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:25:40.295 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:40.295 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:40.296 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:40.296 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:25:40.296 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:25:40.296 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:40.296 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:40.296 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:25:40.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:40.297 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:25:40.297 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:25:40.297 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:25:40.297 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:40.297 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:25:40.298 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-09T18:25:40.783 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:40.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:40.785 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:25:40.821 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:25:40.821 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:25:41.028 INFO:teuthology.orchestra.run.smithi051.stderr:No --period given, using current period=f6258aba-300a-4d5d-ad93-a7c198a0e0a1 2020-11-09T18:25:41.062 INFO:teuthology.orchestra.run.smithi051.stdout:[ 2020-11-09T18:25:41.063 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:25:41.063 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604946340.774139_451.1", 2020-11-09T18:25:41.063 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "2020-11-09T18:25:40.774139Z" 2020-11-09T18:25:41.063 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:25:41.063 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:25:41.063 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:25:41.064 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:25:41.064 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:25:41.064 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:25:41.064 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:25:41.064 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:25:41.064 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:25:41.064 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:25:41.065 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:25:41.065 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:25:41.065 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:25:41.065 INFO:teuthology.orchestra.run.smithi051.stdout:] 2020-11-09T18:25:41.066 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-09T18:25:41.066 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:41.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:41.067 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:25:41.108 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:25:41.109 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:25:41.373 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:25:41.373 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:25:41.374 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:25:41.374 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:25:41.374 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:25:41.374 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:25:41.374 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:25:41.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:41.375 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:25:41.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:41.375 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:25:41.375 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:41.375 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:25:41.375 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:25:41.375 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:41.376 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:25:41.376 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:41.376 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:25:41.376 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:25:41.376 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:41.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:41.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:41.377 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:25:41.377 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:41.377 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:25:41.377 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:25:41.377 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:41.377 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:41.377 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:41.377 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:25:41.378 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:25:41.378 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:41.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:41.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:41.378 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:25:41.378 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:41.378 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:25:41.379 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:25:41.379 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:41.379 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:41.379 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:41.379 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:25:41.379 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:25:41.379 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:41.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:41.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:41.380 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:25:41.380 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:41.380 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:25:41.380 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:25:41.380 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:41.381 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:41.381 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:41.381 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:25:41.381 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:25:41.381 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:41.381 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:41.381 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:25:41.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:41.382 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:25:41.382 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:25:41.382 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:25:41.382 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:41.382 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:25:41.383 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:25:41.799 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2020-11-09T18:25:46.384 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:46.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:46.386 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:25:46.427 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:25:46.427 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:25:46.699 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:25:46.699 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:25:46.700 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:25:46.700 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:25:46.700 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:25:46.700 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:25:46.700 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:25:46.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:46.700 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:25:46.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:46.701 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:25:46.701 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:46.701 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:25:46.701 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:25:46.701 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:46.701 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:25:46.702 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:46.702 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:25:46.702 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:25:46.702 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:46.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:46.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:46.702 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:25:46.703 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:46.703 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:25:46.703 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:25:46.703 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:46.703 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:46.703 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:46.703 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:25:46.704 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:25:46.704 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:46.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:46.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:46.704 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:25:46.704 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:46.704 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:25:46.705 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:25:46.705 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:46.705 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:46.705 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:46.705 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:25:46.705 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:25:46.705 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:46.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:46.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:46.706 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:25:46.706 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:46.706 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:25:46.706 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:25:46.706 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:46.707 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:46.707 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:46.707 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:25:46.707 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:25:46.707 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:46.707 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:46.707 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:25:46.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:46.708 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:25:46.708 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:25:46.708 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:25:46.708 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:46.708 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:25:46.709 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:25:47.405 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~6s 2020-11-09T18:25:51.717 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:51.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:51.718 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:25:51.759 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:25:51.759 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:25:52.025 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:25:52.026 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:25:52.026 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:25:52.026 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:25:52.026 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:25:52.026 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:25:52.027 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:25:52.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:52.027 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:25:52.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:52.027 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:25:52.027 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:52.027 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:25:52.028 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:25:52.028 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:52.028 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:25:52.028 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:52.028 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:25:52.028 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:25:52.028 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:52.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:52.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:52.029 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:25:52.029 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:52.029 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:25:52.029 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:25:52.029 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:52.030 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:52.030 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:52.030 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:25:52.030 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:25:52.030 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:52.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:52.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:52.031 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:25:52.031 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:52.031 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:25:52.031 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:25:52.031 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:52.031 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:52.032 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:52.032 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:25:52.032 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:25:52.032 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:52.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:52.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:52.032 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:25:52.033 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:52.033 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:25:52.033 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:25:52.033 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:52.033 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:52.033 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:52.033 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:25:52.034 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:25:52.034 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:52.034 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:52.034 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:25:52.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:52.034 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:25:52.034 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:25:52.034 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:25:52.035 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:52.035 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:25:52.036 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:25:53.011 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~11s 2020-11-09T18:25:57.037 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:57.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:25:57.039 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:25:57.083 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:25:57.083 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:25:57.351 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:25:57.351 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:25:57.351 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:25:57.351 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:25:57.352 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:25:57.352 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:25:57.352 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:25:57.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:57.352 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:25:57.352 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:57.353 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:25:57.353 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:57.353 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:25:57.353 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:25:57.353 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:57.353 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:25:57.353 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:57.354 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:25:57.354 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:25:57.354 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:57.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:57.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:57.355 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:25:57.355 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:57.355 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:25:57.355 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:25:57.355 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:57.355 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:57.356 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:57.356 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:25:57.356 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:25:57.356 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:57.356 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:57.356 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:57.356 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:25:57.357 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:57.357 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:25:57.357 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:25:57.357 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:57.357 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:57.357 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:57.358 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:25:57.358 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:25:57.358 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:57.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:57.358 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:25:57.358 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:25:57.358 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:25:57.359 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:25:57.359 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:25:57.359 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:25:57.359 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:25:57.359 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:25:57.359 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:25:57.359 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:25:57.360 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:57.360 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:57.360 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:25:57.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:25:57.360 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:25:57.360 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:25:57.360 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:25:57.361 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:25:57.361 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:25:57.362 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:25:58.616 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~17s 2020-11-09T18:26:02.363 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:26:02.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:26:02.364 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:26:02.405 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:26:02.405 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:26:02.670 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:26:02.671 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:26:02.671 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:26:02.671 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:26:02.671 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:26:02.671 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:26:02.671 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:26:02.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:02.672 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:26:02.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:26:02.672 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:26:02.672 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:26:02.672 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:26:02.673 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:26:02.673 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:26:02.673 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:26:02.673 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:26:02.673 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:26:02.673 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:26:02.674 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:02.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:02.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:26:02.674 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:26:02.674 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:26:02.674 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:26:02.674 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:26:02.675 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:26:02.675 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:26:02.675 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:26:02.675 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:26:02.675 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:26:02.675 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:02.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:02.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:26:02.676 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:26:02.676 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:26:02.676 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:26:02.676 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:26:02.677 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:26:02.677 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:26:02.677 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:26:02.677 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:26:02.677 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:26:02.677 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:02.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:02.678 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:26:02.678 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:26:02.678 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:26:02.678 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:26:02.678 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:26:02.678 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:26:02.678 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:26:02.679 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:26:02.679 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:26:02.679 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:26:02.679 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:02.679 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:02.679 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:26:02.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:02.680 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:26:02.680 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:26:02.680 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:26:02.680 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:02.680 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:26:02.682 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:26:04.221 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~22s 2020-11-09T18:26:07.683 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:26:07.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:26:07.684 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:26:07.726 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:26:07.726 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:26:07.986 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:26:07.987 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:26:07.987 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:26:07.988 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:26:07.988 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:26:07.988 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:26:07.988 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:26:07.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:07.989 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:26:07.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:26:07.989 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:26:07.990 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:26:07.990 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:26:07.990 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:26:07.990 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:26:07.990 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:26:07.990 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:26:07.991 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:26:07.991 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:26:07.991 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:07.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:07.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:26:07.991 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:26:07.991 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:26:07.992 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:26:07.992 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:26:07.992 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:26:07.992 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:26:07.992 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:26:07.992 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:26:07.992 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:26:07.993 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:07.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:07.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:26:07.993 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:26:07.993 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:26:07.993 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:26:07.993 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:26:07.994 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:26:07.994 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:26:07.994 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:26:07.994 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:26:07.994 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:26:07.994 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:07.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:07.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:26:07.995 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:26:07.995 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:26:07.995 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:26:07.995 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:26:07.995 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:26:07.995 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:26:07.995 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:26:07.996 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:26:07.996 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:26:07.996 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:07.996 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:07.996 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:26:07.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:07.996 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:26:07.997 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:26:07.997 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:26:07.997 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:07.997 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:26:07.998 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:26:09.826 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~28s 2020-11-09T18:26:12.999 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:26:13.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:26:13.001 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:26:13.042 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:26:13.043 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:26:13.315 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:26:13.315 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:26:13.316 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:26:13.316 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:26:13.316 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:26:13.316 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:26:13.316 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:26:13.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:13.317 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:26:13.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:26:13.317 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:26:13.317 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:26:13.317 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:26:13.318 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:26:13.318 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:26:13.318 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:26:13.318 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:26:13.318 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:26:13.318 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:26:13.318 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:13.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:13.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:26:13.319 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:26:13.319 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:26:13.319 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:26:13.319 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:26:13.319 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:26:13.320 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:26:13.320 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:26:13.320 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:26:13.320 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:26:13.320 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:13.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:13.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:26:13.321 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:26:13.321 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:26:13.321 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:26:13.321 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:26:13.321 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:26:13.321 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:26:13.321 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:26:13.321 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:26:13.322 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:26:13.322 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:13.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:13.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:26:13.322 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:26:13.322 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:26:13.323 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:26:13.323 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:26:13.323 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:26:13.323 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:26:13.323 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:26:13.323 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:26:13.323 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:26:13.324 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:13.324 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:13.324 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:26:13.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:13.324 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:26:13.324 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:26:13.324 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:26:13.325 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:13.325 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:26:13.326 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:26:15.433 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~34s 2020-11-09T18:26:18.327 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:26:18.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:26:18.328 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:26:18.370 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:26:18.370 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:26:18.741 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:26:18.742 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:26:18.742 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:26:18.742 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:26:18.743 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:26:18.743 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:26:18.743 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:26:18.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:18.744 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:26:18.744 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:26:18.744 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:26:18.744 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:26:18.745 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:26:18.745 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:26:18.745 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:26:18.745 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:26:18.746 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:26:18.746 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:26:18.746 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:26:18.746 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:18.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:18.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:26:18.747 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:26:18.748 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:26:18.748 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:26:18.748 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:26:18.749 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:26:18.749 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:26:18.749 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:26:18.749 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:26:18.749 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:26:18.749 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:18.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:18.750 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:26:18.750 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:26:18.750 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:26:18.750 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:26:18.750 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:26:18.750 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:26:18.751 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:26:18.751 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:26:18.751 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:26:18.751 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:26:18.751 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:18.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:18.752 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:26:18.752 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:26:18.752 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:26:18.752 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:26:18.752 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:26:18.752 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:26:18.753 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:26:18.753 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:26:18.753 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:26:18.753 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:26:18.753 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:18.753 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:18.753 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:26:18.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:18.754 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:26:18.754 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:26:18.754 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:26:18.754 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:18.754 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:26:18.755 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:26:21.041 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~39s 2020-11-09T18:26:23.756 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:26:23.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:26:23.758 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:26:23.798 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:26:23.799 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:26:24.071 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:26:24.072 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:26:24.072 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:26:24.072 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:26:24.072 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:26:24.072 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:26:24.072 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:26:24.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:24.073 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:26:24.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:26:24.073 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:26:24.073 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:26:24.073 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:26:24.073 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:26:24.074 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:26:24.074 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:26:24.074 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:26:24.074 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:26:24.075 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:26:24.075 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:24.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:24.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:26:24.076 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:26:24.076 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:26:24.076 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:26:24.076 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:26:24.076 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:26:24.077 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:26:24.077 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:26:24.077 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:26:24.077 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:26:24.077 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:24.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:24.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:26:24.078 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:26:24.078 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:26:24.078 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:26:24.078 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:26:24.078 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:26:24.079 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:26:24.079 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:26:24.079 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:26:24.079 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:26:24.079 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:24.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:24.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:26:24.080 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:26:24.080 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:26:24.080 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:26:24.080 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:26:24.080 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:26:24.080 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:26:24.080 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:26:24.081 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:26:24.081 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:26:24.081 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:24.081 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:24.081 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:26:24.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:24.081 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:26:24.082 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:26:24.082 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:26:24.082 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:24.082 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:26:24.083 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:26:26.648 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~45s 2020-11-09T18:26:29.085 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:26:29.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:26:29.086 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:26:29.125 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:26:29.126 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:26:29.399 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:26:29.399 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:26:29.399 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:26:29.399 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:26:29.399 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:26:29.400 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:26:29.400 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:26:29.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:29.400 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:26:29.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:26:29.400 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:26:29.400 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:26:29.401 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:26:29.401 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:26:29.401 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:26:29.401 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:26:29.401 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:26:29.401 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:26:29.401 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:26:29.402 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:29.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:29.402 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:26:29.402 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:26:29.402 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:26:29.402 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:26:29.402 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:26:29.403 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:26:29.403 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:26:29.403 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:26:29.403 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:26:29.403 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:26:29.403 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:29.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:29.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:26:29.404 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:26:29.404 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:26:29.404 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:26:29.404 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:26:29.404 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:26:29.404 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:26:29.404 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:26:29.405 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:26:29.405 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:26:29.405 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:29.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:29.405 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:26:29.405 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:26:29.406 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:26:29.406 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:26:29.406 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:26:29.406 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:26:29.406 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:26:29.406 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:26:29.406 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:26:29.407 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:26:29.407 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:29.407 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:29.407 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:26:29.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:29.407 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:26:29.407 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:26:29.407 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:26:29.408 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:29.408 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:26:29.409 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:26:32.255 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~50s 2020-11-09T18:26:34.410 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:26:34.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:26:34.411 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:26:34.454 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:26:34.454 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:26:34.721 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:26:34.722 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:26:34.722 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:26:34.722 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:26:34.722 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:26:34.722 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:26:34.722 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:26:34.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:34.723 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:26:34.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:26:34.723 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:26:34.723 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:26:34.723 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:26:34.723 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:26:34.724 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:26:34.724 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:26:34.724 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:26:34.724 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:26:34.724 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:26:34.724 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:34.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:34.725 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:26:34.725 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:26:34.725 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:26:34.725 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:26:34.725 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:26:34.725 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:26:34.725 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:26:34.726 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:26:34.726 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:26:34.726 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:26:34.726 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:34.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:34.726 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:26:34.726 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:26:34.727 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:26:34.727 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:26:34.727 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:26:34.727 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:26:34.727 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:26:34.727 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:26:34.727 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:26:34.727 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:26:34.728 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:34.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:34.728 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:26:34.728 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:26:34.728 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:26:34.728 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:26:34.728 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:26:34.729 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:26:34.729 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:26:34.729 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:26:34.729 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:26:34.729 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:26:34.729 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:34.729 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:34.730 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:26:34.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:34.730 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:26:34.730 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:26:34.730 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:26:34.730 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:34.730 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:26:34.731 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:26:37.865 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~56s 2020-11-09T18:26:39.733 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:26:39.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:26:39.734 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:26:39.773 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:26:39.774 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:26:40.040 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:26:40.040 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:26:40.040 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:26:40.040 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:26:40.040 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:26:40.041 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:26:40.041 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:26:40.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:40.041 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:26:40.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:26:40.041 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:26:40.041 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:26:40.042 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:26:40.042 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:26:40.042 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:26:40.042 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:26:40.042 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:26:40.042 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:26:40.043 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:26:40.043 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:40.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:40.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:26:40.043 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:26:40.043 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:26:40.044 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:26:40.044 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:26:40.044 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:26:40.044 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:26:40.044 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:26:40.044 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:26:40.044 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:26:40.044 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:40.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:40.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:26:40.045 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:26:40.045 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:26:40.045 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:26:40.045 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:26:40.045 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:26:40.046 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:26:40.046 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:26:40.046 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:26:40.046 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:26:40.046 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:40.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:40.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:26:40.047 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:26:40.047 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:26:40.047 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:26:40.047 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:26:40.047 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:26:40.047 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:26:40.047 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:26:40.048 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:26:40.048 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:26:40.048 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:40.048 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:40.048 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:26:40.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:40.048 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:26:40.048 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:26:40.049 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:26:40.049 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:40.049 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:26:40.050 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:26:43.470 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~62s 2020-11-09T18:26:45.051 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:26:45.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:26:45.052 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:26:45.097 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:26:45.097 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:26:45.366 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:26:45.366 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:26:45.366 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:26:45.367 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:26:45.367 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:26:45.367 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:26:45.367 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:26:45.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:45.368 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:26:45.368 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:26:45.368 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:26:45.368 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:26:45.368 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:26:45.368 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:26:45.369 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:26:45.369 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:26:45.369 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:26:45.369 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:26:45.369 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:26:45.369 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:45.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:45.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:26:45.370 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:26:45.370 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:26:45.370 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:26:45.370 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:26:45.370 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:26:45.370 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:26:45.371 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:26:45.371 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:26:45.371 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:26:45.371 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:45.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:45.371 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:26:45.372 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:26:45.372 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:26:45.372 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:26:45.372 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:26:45.372 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:26:45.372 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:26:45.373 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:26:45.373 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:26:45.373 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:26:45.373 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:45.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:45.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:26:45.374 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:26:45.374 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:26:45.374 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:26:45.374 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:26:45.375 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:26:45.375 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:26:45.375 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:26:45.375 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:26:45.375 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:26:45.375 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:45.375 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:45.376 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:26:45.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:45.376 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:26:45.376 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:26:45.376 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:26:45.376 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:45.376 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:26:45.377 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:26:49.076 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~67s 2020-11-09T18:26:50.378 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:26:50.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:26:50.380 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:26:50.422 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:26:50.423 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:26:50.688 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:26:50.688 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:26:50.688 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:26:50.688 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:26:50.688 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:26:50.689 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:26:50.689 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:26:50.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:50.689 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:26:50.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:26:50.689 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:26:50.689 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:26:50.690 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:26:50.690 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:26:50.690 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:26:50.690 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:26:50.690 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:26:50.690 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:26:50.691 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:26:50.691 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:50.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:50.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:26:50.691 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:26:50.691 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:26:50.691 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:26:50.691 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:26:50.692 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:26:50.692 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:26:50.692 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:26:50.692 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:26:50.692 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:26:50.692 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:50.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:50.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:26:50.693 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:26:50.693 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:26:50.693 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:26:50.693 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:26:50.693 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:26:50.693 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:26:50.694 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:26:50.694 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:26:50.694 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:26:50.694 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:50.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:50.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:26:50.694 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:26:50.695 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:26:50.695 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:26:50.695 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:26:50.695 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:26:50.695 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:26:50.695 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:26:50.695 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:26:50.695 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:26:50.696 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:50.696 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:50.696 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:26:50.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:50.696 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:26:50.696 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:26:50.696 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:26:50.697 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:50.697 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:26:50.698 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:26:54.682 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~73s 2020-11-09T18:26:55.699 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:26:55.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:26:55.701 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:26:55.745 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:26:55.745 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:26:56.007 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:26:56.007 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:26:56.007 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:26:56.008 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:26:56.008 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:26:56.008 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:26:56.008 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:26:56.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:56.008 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:26:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:26:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:26:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:26:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:26:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:26:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:26:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:26:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:26:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:26:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:26:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:26:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:26:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:26:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:26:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:26:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:26:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:26:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:26:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:26:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:26:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:26:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:26:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:26:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:26:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:26:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:26:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:26:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:26:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:26:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:26:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:26:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:26:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:26:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:26:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:26:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:26:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:26:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:26:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:26:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:26:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:26:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:26:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:26:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:26:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:26:56.017 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:26:56.017 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:26:56.018 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:27:00.289 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~78s 2020-11-09T18:27:01.019 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:27:01.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:27:01.021 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:27:01.061 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:27:01.062 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:27:01.331 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:27:01.332 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:27:01.332 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:27:01.332 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:27:01.332 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:27:01.332 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:27:01.332 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:27:01.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:01.333 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:27:01.333 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:27:01.333 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:27:01.333 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:27:01.333 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:27:01.334 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:27:01.334 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:27:01.334 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:27:01.334 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:27:01.334 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:27:01.334 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:27:01.334 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:01.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:01.335 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:27:01.335 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:27:01.335 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:27:01.335 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:27:01.335 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:27:01.335 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:27:01.336 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:27:01.336 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:27:01.336 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:27:01.336 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:27:01.336 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:01.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:01.336 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:27:01.336 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:27:01.337 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:27:01.337 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:27:01.337 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:27:01.337 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:27:01.337 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:27:01.337 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:27:01.337 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:27:01.338 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:27:01.338 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:01.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:01.338 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:27:01.338 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:27:01.338 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:27:01.338 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:27:01.339 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:27:01.339 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:27:01.339 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:27:01.339 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:27:01.339 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:27:01.339 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:27:01.339 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:01.339 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:01.340 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:27:01.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:01.340 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:27:01.340 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:27:01.340 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:27:01.340 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:01.340 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:27:01.341 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:27:05.894 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~84s 2020-11-09T18:27:06.342 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:27:06.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:27:06.344 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:27:06.387 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:27:06.387 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:27:06.656 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:27:06.657 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:27:06.657 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:27:06.657 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:27:06.657 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:27:06.657 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:27:06.657 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:27:06.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:06.658 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:27:06.658 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:27:06.658 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:27:06.658 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:27:06.658 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:27:06.658 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:27:06.659 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:27:06.659 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:27:06.659 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:27:06.659 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:27:06.659 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:27:06.659 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:06.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:06.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:27:06.660 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:27:06.660 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:27:06.660 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:27:06.660 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:27:06.660 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:27:06.660 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:27:06.660 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:27:06.661 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:27:06.661 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:27:06.661 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:06.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:06.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:27:06.661 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:27:06.661 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:27:06.662 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:27:06.662 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:27:06.662 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:27:06.662 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:27:06.662 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:27:06.662 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:27:06.662 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:27:06.663 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:06.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:06.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:27:06.663 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:27:06.663 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:27:06.663 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:27:06.663 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:27:06.664 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:27:06.664 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:27:06.664 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:27:06.664 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:27:06.664 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:27:06.664 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:06.664 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:06.664 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:27:06.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:06.665 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:27:06.665 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:27:06.665 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:27:06.665 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:06.665 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:27:06.666 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:27:11.500 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~90s 2020-11-09T18:27:11.667 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:27:11.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:27:11.669 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:27:11.708 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:27:11.709 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:27:11.982 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:27:11.982 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:27:11.982 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:27:11.983 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:27:11.983 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:27:11.983 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:27:11.983 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:27:11.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:11.983 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:27:11.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:27:11.984 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:27:11.984 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:27:11.984 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:27:11.984 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:27:11.984 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:27:11.984 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:27:11.984 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:27:11.985 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:27:11.985 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:27:11.985 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:11.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:11.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:27:11.985 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:27:11.985 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:27:11.985 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:27:11.986 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:27:11.986 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:27:11.986 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:27:11.986 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:27:11.986 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:27:11.986 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:27:11.986 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:11.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:11.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:27:11.987 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:27:11.987 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:27:11.987 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:27:11.987 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:27:11.987 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:27:11.988 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:27:11.988 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:27:11.988 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:27:11.988 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:27:11.988 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:11.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:11.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:27:11.988 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:27:11.989 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:27:11.989 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:27:11.989 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:27:11.989 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:27:11.989 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:27:11.989 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:27:11.989 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:27:11.990 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:27:11.990 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:11.990 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:11.990 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:27:11.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:11.990 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:27:11.990 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:27:11.991 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:27:11.991 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:11.991 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:27:11.992 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:27:16.993 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:27:16.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:27:16.994 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:27:17.041 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:27:17.041 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:27:17.108 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~95s 2020-11-09T18:27:17.305 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:27:17.305 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:27:17.306 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:27:17.306 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:27:17.306 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:27:17.306 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:27:17.306 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:27:17.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:17.306 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:27:17.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:27:17.307 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:27:17.307 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:27:17.307 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:27:17.307 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:27:17.307 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:27:17.307 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:27:17.308 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:27:17.308 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:27:17.308 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:27:17.308 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:17.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:17.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:27:17.308 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:27:17.309 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:27:17.309 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:27:17.309 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:27:17.309 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:27:17.309 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:27:17.309 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:27:17.309 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:27:17.310 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:27:17.310 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:17.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:17.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:27:17.310 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:27:17.311 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:27:17.311 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:27:17.311 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:27:17.311 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:27:17.311 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:27:17.311 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:27:17.312 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:27:17.312 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:27:17.312 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:17.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:17.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:27:17.312 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:27:17.312 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:27:17.313 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:27:17.313 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:27:17.313 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:27:17.313 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:27:17.313 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:27:17.313 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:27:17.313 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:27:17.313 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:17.314 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:17.314 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:27:17.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:17.314 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:27:17.314 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:27:17.314 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:27:17.315 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:17.315 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:27:17.316 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:27:22.318 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:27:22.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:27:22.319 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:27:22.361 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:27:22.362 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:27:22.634 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:27:22.634 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:27:22.635 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:27:22.635 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:27:22.635 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:27:22.635 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:27:22.635 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:27:22.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:22.635 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:27:22.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:27:22.636 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:27:22.636 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:27:22.636 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:27:22.636 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:27:22.636 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:27:22.636 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:27:22.637 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:27:22.637 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:27:22.637 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:27:22.637 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:22.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:22.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:27:22.637 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:27:22.638 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:27:22.638 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:27:22.638 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:27:22.638 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:27:22.638 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:27:22.638 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:27:22.638 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:27:22.639 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:27:22.639 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:22.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:22.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:27:22.639 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:27:22.639 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:27:22.639 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:27:22.639 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:27:22.640 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:27:22.640 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:27:22.640 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:27:22.640 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:27:22.640 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:27:22.640 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:22.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:22.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:27:22.641 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:27:22.641 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:27:22.641 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:27:22.641 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:27:22.641 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:27:22.641 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:27:22.642 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:27:22.642 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:27:22.642 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:27:22.642 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:22.642 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:22.642 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:27:22.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:22.642 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:27:22.643 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:27:22.643 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:27:22.643 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:22.643 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:27:22.644 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:27:22.713 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~101s 2020-11-09T18:27:27.645 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:27:27.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:27:27.664 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:27:27.705 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:27:27.706 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:27:28.071 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:27:28.072 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:27:28.072 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:27:28.072 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:27:28.072 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:27:28.073 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:27:28.073 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:27:28.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:28.073 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:27:28.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:27:28.073 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:27:28.074 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:27:28.074 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:27:28.074 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:27:28.074 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:27:28.074 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:27:28.075 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:27:28.075 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:27:28.075 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:27:28.075 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:28.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:28.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:27:28.076 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:27:28.076 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:27:28.076 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:27:28.076 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:27:28.076 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:27:28.077 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:27:28.077 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:27:28.077 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:27:28.077 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:27:28.077 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:28.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:28.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:27:28.079 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:27:28.079 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:27:28.079 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:27:28.080 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:27:28.080 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:27:28.080 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:27:28.080 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:27:28.080 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:27:28.080 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:27:28.080 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:28.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:28.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:27:28.081 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:27:28.081 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:27:28.081 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:27:28.081 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:27:28.081 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:27:28.082 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:27:28.082 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:27:28.082 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:27:28.082 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:27:28.083 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:28.083 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:28.083 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:27:28.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:28.083 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:27:28.083 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:27:28.084 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:27:28.084 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:28.084 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:27:28.085 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:27:28.319 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~107s 2020-11-09T18:27:33.086 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:27:33.087 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:27:33.087 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:27:33.130 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:27:33.131 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:27:33.495 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:27:33.496 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:27:33.496 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:27:33.496 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:27:33.496 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:27:33.496 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:27:33.496 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:27:33.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:33.497 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:27:33.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:27:33.497 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:27:33.497 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:27:33.497 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:27:33.497 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:27:33.498 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:27:33.498 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:27:33.498 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:27:33.498 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:27:33.498 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:27:33.498 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:33.498 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:33.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:27:33.499 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:27:33.499 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:27:33.499 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:27:33.499 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:27:33.499 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:27:33.499 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:27:33.500 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:27:33.500 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:27:33.500 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:27:33.500 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:33.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:33.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:27:33.500 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:27:33.500 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:27:33.501 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:27:33.501 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:27:33.501 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:27:33.501 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:27:33.501 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:27:33.501 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:27:33.501 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:27:33.502 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:33.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:33.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:27:33.502 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:27:33.502 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:27:33.502 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:27:33.502 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:27:33.503 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:27:33.503 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:27:33.503 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:27:33.503 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:27:33.503 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:27:33.503 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:33.503 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:33.504 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:27:33.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:33.504 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:27:33.504 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:27:33.505 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:27:33.505 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:33.505 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:27:33.506 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:27:33.925 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~112s 2020-11-09T18:27:38.507 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:27:38.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:27:38.509 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:27:38.549 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:27:38.549 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:27:38.820 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:27:38.820 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:27:38.820 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:27:38.820 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:27:38.820 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:27:38.821 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:27:38.821 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:27:38.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:38.821 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:27:38.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:27:38.821 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:27:38.822 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:27:38.822 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:27:38.822 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:27:38.822 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:27:38.822 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:27:38.822 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:27:38.822 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:27:38.823 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:27:38.823 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:38.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:38.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:27:38.823 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:27:38.823 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:27:38.823 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:27:38.824 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:27:38.824 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:27:38.824 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:27:38.824 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:27:38.824 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:27:38.824 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:27:38.825 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:38.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:38.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:27:38.825 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:27:38.825 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:27:38.825 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:27:38.825 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:27:38.826 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:27:38.826 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:27:38.826 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:27:38.826 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:27:38.826 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:27:38.826 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:38.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:38.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:27:38.827 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:27:38.827 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:27:38.827 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:27:38.827 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:27:38.827 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:27:38.828 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:27:38.828 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:27:38.828 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:27:38.828 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:27:38.828 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:38.828 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:38.829 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:27:38.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:38.829 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:27:38.829 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:27:38.829 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:27:38.829 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:38.829 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:27:38.830 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:27:39.532 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~118s 2020-11-09T18:27:43.832 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:27:43.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:27:43.833 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:27:43.877 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:27:43.877 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:27:44.144 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:27:44.144 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:27:44.144 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:27:44.145 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:27:44.145 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:27:44.145 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:27:44.145 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:27:44.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:44.145 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:27:44.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:27:44.146 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:27:44.146 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:27:44.146 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:27:44.146 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:27:44.146 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:27:44.146 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:27:44.146 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:27:44.147 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:27:44.147 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:27:44.147 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:44.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:44.147 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:27:44.147 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:27:44.147 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:27:44.148 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:27:44.148 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:27:44.148 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:27:44.148 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:27:44.148 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:27:44.148 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:27:44.148 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:27:44.149 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:44.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:44.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:27:44.149 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:27:44.149 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:27:44.149 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:27:44.149 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:27:44.150 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:27:44.150 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:27:44.150 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:27:44.150 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:27:44.150 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:27:44.150 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:44.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:44.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:27:44.151 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:27:44.151 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:27:44.151 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:27:44.151 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:27:44.151 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:27:44.151 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:27:44.151 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:27:44.152 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:27:44.152 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:27:44.152 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:44.152 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:44.152 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:27:44.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:44.152 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:27:44.153 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:27:44.153 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:27:44.153 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:44.153 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:27:44.154 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:27:45.137 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~123s 2020-11-09T18:27:49.155 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:27:49.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:27:49.157 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:27:49.196 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:27:49.197 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:27:49.468 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:27:49.469 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:27:49.469 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:27:49.469 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:27:49.469 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:27:49.469 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:27:49.470 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:27:49.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:49.470 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:27:49.470 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:27:49.470 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:27:49.470 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:27:49.471 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:27:49.471 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:27:49.471 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:27:49.471 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:27:49.471 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:27:49.471 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:27:49.471 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:27:49.471 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:49.472 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:49.472 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:27:49.472 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:27:49.472 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:27:49.472 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:27:49.472 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:27:49.472 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:27:49.473 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:27:49.473 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:27:49.473 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:27:49.473 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:27:49.473 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:49.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:49.473 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:27:49.474 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:27:49.474 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:27:49.474 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:27:49.474 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:27:49.474 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:27:49.474 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:27:49.474 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:27:49.475 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:27:49.475 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:27:49.475 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:49.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:49.475 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:27:49.475 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:27:49.475 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:27:49.476 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:27:49.476 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:27:49.476 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:27:49.476 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:27:49.476 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:27:49.476 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:27:49.476 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:27:49.476 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:49.477 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:49.477 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:27:49.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:49.477 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:27:49.477 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:27:49.477 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:27:49.477 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:49.478 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:27:49.479 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:27:50.744 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~129s 2020-11-09T18:27:54.480 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:27:54.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:27:54.481 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:27:54.521 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:27:54.521 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:27:54.793 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:27:54.794 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:27:54.794 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:27:54.794 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:27:54.794 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:27:54.794 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:27:54.794 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:27:54.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:54.795 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:27:54.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:27:54.795 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:27:54.795 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:27:54.795 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:27:54.795 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:27:54.796 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:27:54.796 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:27:54.796 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:27:54.796 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:27:54.796 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:27:54.796 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:54.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:54.797 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:27:54.797 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:27:54.797 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:27:54.797 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:27:54.797 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:27:54.797 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:27:54.797 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:27:54.797 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:27:54.798 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:27:54.798 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:27:54.798 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:54.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:54.798 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:27:54.798 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:27:54.798 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:27:54.799 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:27:54.799 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:27:54.799 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:27:54.799 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:27:54.799 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:27:54.799 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:27:54.799 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:27:54.800 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:54.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:54.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:27:54.800 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:27:54.800 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:27:54.800 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:27:54.800 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:27:54.800 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:27:54.801 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:27:54.801 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:27:54.801 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:27:54.801 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:27:54.801 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:54.801 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:54.801 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:27:54.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:27:54.802 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:27:54.802 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:27:54.802 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:27:54.802 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:27:54.802 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:27:54.803 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:27:56.351 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~135s 2020-11-09T18:27:59.805 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:27:59.806 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:27:59.806 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:27:59.849 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:27:59.849 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:28:00.119 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:28:00.119 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:28:00.120 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:28:00.120 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:28:00.120 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:28:00.120 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:28:00.120 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:28:00.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:00.121 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:28:00.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:00.121 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:28:00.121 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:00.121 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:00.122 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:28:00.122 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:00.122 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:28:00.122 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:00.122 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:28:00.122 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:28:00.122 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:00.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:00.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:00.123 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:28:00.123 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:00.123 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:00.123 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:28:00.123 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:00.124 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:28:00.124 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:00.124 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:28:00.124 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:28:00.124 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:00.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:00.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:00.124 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:28:00.125 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:00.125 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:00.125 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:28:00.125 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:00.125 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:28:00.125 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:00.126 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:28:00.126 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:28:00.126 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:00.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:00.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:00.126 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:28:00.126 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:00.127 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:00.127 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:28:00.127 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:00.127 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:28:00.127 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:00.127 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:28:00.127 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:28:00.127 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:00.128 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:00.128 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:28:00.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:00.128 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:28:00.128 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:28:00.129 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:28:00.129 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:00.129 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:28:00.130 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:28:01.957 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~140s 2020-11-09T18:28:05.132 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:28:05.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:28:05.134 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:28:05.175 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:28:05.176 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:28:05.436 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:28:05.437 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:28:05.437 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:28:05.437 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:28:05.437 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:28:05.437 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:28:05.438 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:28:05.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:05.438 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:28:05.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:05.438 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:28:05.438 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:05.439 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:05.439 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:28:05.439 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:05.439 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:28:05.439 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:05.439 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:28:05.439 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:28:05.440 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:05.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:05.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:05.440 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:28:05.440 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:05.440 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:05.440 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:28:05.440 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:05.441 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:28:05.441 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:05.441 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:28:05.441 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:28:05.441 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:05.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:05.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:05.442 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:28:05.442 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:05.442 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:05.442 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:28:05.442 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:05.442 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:28:05.442 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:05.443 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:28:05.443 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:28:05.443 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:05.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:05.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:05.443 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:28:05.443 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:05.444 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:05.444 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:28:05.444 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:05.444 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:28:05.444 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:05.444 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:28:05.444 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:28:05.444 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:05.445 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:05.445 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:28:05.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:05.445 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:28:05.445 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:28:05.445 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:28:05.445 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:05.446 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:28:05.447 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:28:07.564 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~146s 2020-11-09T18:28:10.448 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:28:10.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:28:10.449 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:28:10.489 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:28:10.490 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:28:10.861 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:28:10.862 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:28:10.862 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:28:10.862 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:28:10.862 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:28:10.862 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:28:10.863 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:28:10.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:10.863 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:28:10.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:10.863 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:28:10.863 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:10.863 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:10.864 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:28:10.864 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:10.864 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:28:10.864 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:10.864 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:28:10.864 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:28:10.864 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:10.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:10.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:10.865 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:28:10.865 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:10.865 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:10.865 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:28:10.865 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:10.865 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:28:10.866 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:10.866 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:28:10.866 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:28:10.866 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:10.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:10.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:10.866 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:28:10.867 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:10.867 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:10.867 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:28:10.867 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:10.867 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:28:10.867 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:10.867 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:28:10.867 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:28:10.868 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:10.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:10.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:10.868 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:28:10.868 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:10.868 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:10.868 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:28:10.869 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:10.869 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:28:10.869 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:10.869 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:28:10.869 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:28:10.869 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:10.869 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:10.870 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:28:10.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:10.871 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:28:10.871 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:28:10.871 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:28:10.871 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:10.872 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:28:10.873 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:28:13.171 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~151s 2020-11-09T18:28:15.874 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:28:15.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:28:15.875 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:28:15.916 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:28:15.916 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:28:16.189 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:28:16.190 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:28:16.190 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:28:16.190 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:28:16.191 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:28:16.191 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:28:16.191 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:28:16.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:16.192 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:28:16.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:16.192 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:28:16.193 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:16.193 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:16.193 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:28:16.194 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:16.194 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:28:16.194 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:16.194 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:28:16.195 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:28:16.195 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:16.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:16.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:16.196 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:28:16.196 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:16.196 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:16.197 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:28:16.197 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:16.197 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:28:16.197 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:16.198 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:28:16.198 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:28:16.198 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:16.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:16.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:16.199 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:28:16.199 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:16.200 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:16.200 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:28:16.200 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:16.200 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:28:16.201 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:16.201 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:28:16.201 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:28:16.201 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:16.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:16.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:16.202 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:28:16.202 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:16.203 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:16.203 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:28:16.203 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:16.204 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:28:16.204 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:16.204 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:28:16.204 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:28:16.205 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:16.205 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:16.205 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:28:16.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:16.206 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:28:16.206 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:28:16.206 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:28:16.207 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:16.207 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:28:16.208 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:28:18.776 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~157s 2020-11-09T18:28:21.210 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:28:21.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:28:21.211 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:28:21.252 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:28:21.252 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:28:21.518 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:28:21.519 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:28:21.519 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:28:21.519 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:28:21.519 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:28:21.519 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:28:21.520 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:28:21.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:21.520 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:28:21.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:21.520 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:28:21.520 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:21.520 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:21.521 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:28:21.521 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:21.521 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:28:21.521 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:21.521 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:28:21.521 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:28:21.521 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:21.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:21.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:21.522 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:28:21.522 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:21.522 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:21.522 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:28:21.522 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:21.523 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:28:21.523 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:21.523 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:28:21.523 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:28:21.523 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:21.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:21.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:21.524 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:28:21.524 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:21.524 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:21.524 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:28:21.524 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:21.524 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:28:21.524 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:21.524 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:28:21.525 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:28:21.525 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:21.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:21.525 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:21.525 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:28:21.525 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:21.525 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:21.526 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:28:21.526 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:21.526 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:28:21.526 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:21.526 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:28:21.526 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:28:21.526 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:21.527 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:21.527 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:28:21.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:21.527 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:28:21.527 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:28:21.527 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:28:21.527 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:21.528 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:28:21.528 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:28:24.382 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~163s 2020-11-09T18:28:26.530 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:28:26.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:28:26.531 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:28:26.572 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:28:26.572 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:28:26.844 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:28:26.845 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:28:26.845 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:28:26.845 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:28:26.845 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:28:26.845 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:28:26.846 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:28:26.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:26.846 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:28:26.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:26.846 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:28:26.846 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:26.847 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:26.847 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:28:26.847 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:26.847 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:28:26.847 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:26.847 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:28:26.847 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:28:26.848 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:26.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:26.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:26.848 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:28:26.848 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:26.848 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:26.848 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:28:26.849 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:26.849 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:28:26.849 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:26.849 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:28:26.849 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:28:26.849 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:26.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:26.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:26.850 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:28:26.850 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:26.850 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:26.850 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:28:26.850 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:26.851 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:28:26.851 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:26.851 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:28:26.851 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:28:26.851 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:26.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:26.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:26.852 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:28:26.852 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:26.852 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:26.852 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:28:26.852 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:26.852 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:28:26.852 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:26.853 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:28:26.853 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:28:26.853 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:26.853 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:26.853 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:28:26.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:26.854 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:28:26.854 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:28:26.854 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:28:26.854 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:26.854 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:28:26.855 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:28:29.988 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~168s 2020-11-09T18:28:31.856 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:28:31.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:28:31.859 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:28:31.900 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:28:31.901 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:28:32.169 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:28:32.170 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:28:32.170 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:28:32.170 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:28:32.170 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:28:32.170 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:28:32.171 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:28:32.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:32.171 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:28:32.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:32.171 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:28:32.171 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:32.171 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:32.172 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:28:32.172 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:32.172 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:28:32.172 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:32.172 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:28:32.172 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:28:32.172 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:32.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:32.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:32.173 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:28:32.173 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:32.173 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:32.173 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:28:32.173 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:32.174 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:28:32.174 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:32.174 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:28:32.174 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:28:32.174 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:32.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:32.174 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:32.175 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:28:32.175 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:32.175 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:32.175 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:28:32.175 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:32.175 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:28:32.175 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:32.176 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:28:32.176 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:28:32.176 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:32.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:32.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:32.176 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:28:32.176 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:32.176 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:32.177 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:28:32.177 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:32.177 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:28:32.177 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:32.177 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:28:32.177 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:28:32.177 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:32.178 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:32.178 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:28:32.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:32.178 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:28:32.178 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:28:32.178 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:28:32.178 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:32.179 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:28:32.179 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:28:35.596 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~174s 2020-11-09T18:28:37.181 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:28:37.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:28:37.182 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:28:37.224 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:28:37.224 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:28:37.497 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:28:37.498 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:28:37.498 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:28:37.498 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:28:37.498 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:28:37.498 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:28:37.498 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:28:37.498 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:37.499 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:28:37.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:37.499 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:28:37.499 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:37.499 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:37.499 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:28:37.499 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:37.500 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:28:37.500 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:37.500 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:28:37.500 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:28:37.500 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:37.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:37.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:37.501 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:28:37.501 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:37.501 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:37.501 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:28:37.501 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:37.501 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:28:37.501 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:37.502 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:28:37.502 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:28:37.502 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:37.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:37.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:37.502 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:28:37.502 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:37.503 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:37.503 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:28:37.503 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:37.503 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:28:37.503 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:37.503 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:28:37.504 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:28:37.504 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:37.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:37.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:37.504 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:28:37.504 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:37.504 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:37.504 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:28:37.505 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:37.505 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:28:37.505 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:37.505 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:28:37.505 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:28:37.505 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:37.505 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:37.506 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:28:37.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:37.506 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:28:37.506 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:28:37.506 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:28:37.506 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:37.506 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:28:37.507 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:28:41.203 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~179s 2020-11-09T18:28:42.508 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:28:42.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:28:42.510 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:28:42.553 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:28:42.553 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:28:42.826 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:28:42.826 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:28:42.826 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:28:42.827 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:28:42.827 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:28:42.827 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:28:42.827 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:28:42.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:42.827 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:28:42.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:42.828 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:28:42.828 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:42.828 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:42.828 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:28:42.828 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:42.828 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:28:42.828 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:42.829 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:28:42.829 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:28:42.829 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:42.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:42.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:42.829 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:28:42.829 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:42.829 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:42.830 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:28:42.830 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:42.830 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:28:42.830 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:42.830 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:28:42.830 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:28:42.830 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:42.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:42.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:42.831 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:28:42.831 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:42.831 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:42.831 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:28:42.831 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:42.832 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:28:42.832 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:42.832 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:28:42.832 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:28:42.832 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:42.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:42.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:42.833 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:28:42.833 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:42.833 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:42.833 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:28:42.833 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:42.833 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:28:42.833 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:42.833 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:28:42.834 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:28:42.834 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:42.834 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:42.834 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:28:42.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:42.834 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:28:42.834 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:28:42.835 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:28:42.835 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:42.835 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:28:42.836 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:28:46.810 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~185s 2020-11-09T18:28:47.838 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:28:47.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:28:47.839 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:28:47.880 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:28:47.880 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:28:48.143 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:28:48.143 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:28:48.144 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:28:48.144 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:28:48.144 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:28:48.144 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:28:48.144 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:28:48.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:48.144 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:28:48.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:48.145 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:28:48.145 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:48.145 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:48.145 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:28:48.145 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:48.145 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:28:48.146 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:48.146 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:28:48.146 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:28:48.146 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:48.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:48.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:48.147 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:28:48.147 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:48.147 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:48.147 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:28:48.147 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:48.147 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:28:48.147 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:48.147 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:28:48.148 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:28:48.148 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:48.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:48.148 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:48.148 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:28:48.148 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:48.148 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:48.149 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:28:48.149 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:48.149 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:28:48.149 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:48.149 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:28:48.150 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:28:48.150 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:48.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:48.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:48.150 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:28:48.150 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:48.150 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:48.150 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:28:48.151 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:48.151 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:28:48.151 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:48.151 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:28:48.151 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:28:48.151 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:48.151 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:48.152 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:28:48.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:48.152 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:28:48.152 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:28:48.152 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:28:48.152 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:48.152 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:28:48.153 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:28:52.416 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~191s 2020-11-09T18:28:53.155 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:28:53.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:28:53.156 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:28:53.198 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:28:53.198 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:28:53.474 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:28:53.475 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:28:53.475 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:28:53.475 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:28:53.475 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:28:53.475 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:28:53.475 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:28:53.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:53.476 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:28:53.476 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:53.476 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:28:53.476 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:53.476 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:53.477 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:28:53.477 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:53.477 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:28:53.477 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:53.477 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:28:53.477 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:28:53.478 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:53.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:53.478 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:53.478 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:28:53.478 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:53.478 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:53.479 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:28:53.479 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:53.479 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:28:53.479 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:53.479 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:28:53.479 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:28:53.480 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:53.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:53.480 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:53.480 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:28:53.480 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:53.480 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:53.481 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:28:53.481 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:53.481 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:28:53.481 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:53.481 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:28:53.481 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:28:53.482 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:53.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:53.482 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:53.482 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:28:53.482 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:53.482 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:53.483 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:28:53.483 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:53.483 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:28:53.483 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:53.483 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:28:53.483 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:28:53.484 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:53.484 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:53.484 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:28:53.484 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:53.484 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:28:53.484 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:28:53.485 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:28:53.485 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:53.485 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:28:53.486 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:28:58.022 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~196s 2020-11-09T18:28:58.487 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:28:58.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:28:58.488 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:28:58.530 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:28:58.530 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:28:58.794 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:28:58.795 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:28:58.795 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:28:58.795 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:28:58.795 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:28:58.795 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:28:58.796 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:28:58.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:58.796 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:28:58.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:58.796 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:28:58.796 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:58.796 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:58.797 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:28:58.797 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:58.797 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:28:58.797 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:58.797 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:28:58.797 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:28:58.797 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:58.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:58.798 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:58.798 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:28:58.798 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:58.798 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:58.798 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:28:58.798 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:58.799 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:28:58.799 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:58.799 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:28:58.799 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:28:58.799 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:58.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:58.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:58.799 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:28:58.800 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:58.800 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:58.800 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:28:58.800 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:58.800 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:28:58.800 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:58.800 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:28:58.801 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:28:58.801 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:58.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:58.801 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:28:58.801 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:28:58.801 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:28:58.801 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:28:58.802 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:28:58.802 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:28:58.802 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:28:58.802 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:28:58.802 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:28:58.802 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:28:58.802 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:58.802 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:58.803 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:28:58.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:28:58.803 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:28:58.803 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:28:58.803 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:28:58.803 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:28:58.803 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:28:58.804 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:29:03.628 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~202s 2020-11-09T18:29:03.805 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:29:03.806 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:29:03.807 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:29:03.849 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:29:03.849 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:29:04.121 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:29:04.121 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:29:04.121 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:29:04.121 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:29:04.122 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:29:04.122 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:29:04.122 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:29:04.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:04.122 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:29:04.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:29:04.123 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:29:04.123 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:29:04.123 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:29:04.123 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:29:04.123 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:29:04.123 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:29:04.123 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:29:04.124 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:29:04.124 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:29:04.124 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:04.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:04.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:29:04.124 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:29:04.124 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:29:04.124 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:29:04.125 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:29:04.125 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:29:04.125 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:29:04.125 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:29:04.125 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:29:04.125 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:29:04.125 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:04.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:04.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:29:04.126 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:29:04.126 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:29:04.126 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:29:04.126 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:29:04.127 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:29:04.127 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:29:04.127 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:29:04.127 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:29:04.127 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:29:04.127 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:04.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:04.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:29:04.128 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:29:04.128 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:29:04.128 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:29:04.128 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:29:04.128 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:29:04.128 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:29:04.128 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:29:04.129 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:29:04.129 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:29:04.129 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:04.129 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:04.129 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:29:04.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:04.129 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:29:04.130 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:29:04.130 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:29:04.130 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:04.130 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:29:04.131 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:29:09.132 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:29:09.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:29:09.133 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:29:09.174 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:29:09.175 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:29:09.235 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~207s 2020-11-09T18:29:09.547 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:29:09.548 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:29:09.548 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:29:09.548 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:29:09.548 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:29:09.548 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:29:09.549 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:29:09.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:09.549 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:29:09.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:29:09.549 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:29:09.549 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:29:09.549 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:29:09.550 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:29:09.550 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:29:09.550 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:29:09.550 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:29:09.550 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:29:09.550 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:29:09.551 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:09.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:09.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:29:09.551 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:29:09.551 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:29:09.551 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:29:09.551 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:29:09.551 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:29:09.552 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:29:09.552 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:29:09.552 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:29:09.552 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:29:09.552 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:09.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:09.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:29:09.553 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:29:09.553 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:29:09.553 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:29:09.553 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:29:09.553 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:29:09.553 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:29:09.553 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:29:09.554 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:29:09.554 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:29:09.554 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:09.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:09.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:29:09.554 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:29:09.554 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:29:09.555 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:29:09.555 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:29:09.555 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:29:09.555 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:29:09.555 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:29:09.555 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:29:09.555 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:29:09.555 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:09.556 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:09.556 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:29:09.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:09.556 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:29:09.556 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:29:09.556 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:29:09.556 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:09.557 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:29:09.557 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:29:14.559 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:29:14.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:29:14.560 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:29:14.602 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:29:14.603 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:29:14.841 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~213s 2020-11-09T18:29:14.872 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:29:14.872 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:29:14.872 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:29:14.872 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:29:14.873 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:29:14.873 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:29:14.873 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:29:14.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:14.873 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:29:14.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:29:14.874 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:29:14.874 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:29:14.874 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:29:14.874 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:29:14.874 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:29:14.875 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:29:14.875 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:29:14.875 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:29:14.875 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:29:14.875 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:14.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:14.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:29:14.876 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:29:14.876 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:29:14.876 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:29:14.876 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:29:14.876 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:29:14.876 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:29:14.877 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:29:14.877 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:29:14.877 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:29:14.877 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:14.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:14.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:29:14.877 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:29:14.878 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:29:14.883 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:29:14.883 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:29:14.883 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:29:14.883 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:29:14.883 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:29:14.884 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:29:14.884 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:29:14.884 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:14.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:14.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:29:14.884 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:29:14.884 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:29:14.885 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:29:14.885 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:29:14.885 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:29:14.885 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:29:14.885 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:29:14.885 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:29:14.885 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:29:14.885 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:14.886 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:14.886 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:29:14.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:14.886 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:29:14.886 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:29:14.886 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:29:14.886 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:14.887 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:29:14.888 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:29:19.889 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:29:19.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:29:19.890 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:29:19.932 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:29:19.932 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:29:20.200 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:29:20.200 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:29:20.201 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:29:20.201 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:29:20.201 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:29:20.201 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:29:20.202 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:29:20.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:20.202 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:29:20.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:29:20.202 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:29:20.202 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:29:20.203 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:29:20.203 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:29:20.203 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:29:20.203 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:29:20.203 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:29:20.203 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:29:20.203 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:29:20.204 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:20.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:20.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:29:20.204 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:29:20.204 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:29:20.204 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:29:20.204 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:29:20.205 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:29:20.205 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:29:20.205 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:29:20.205 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:29:20.205 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:29:20.205 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:20.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:20.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:29:20.206 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:29:20.206 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:29:20.206 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:29:20.206 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:29:20.206 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:29:20.206 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:29:20.206 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:29:20.207 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:29:20.207 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:29:20.207 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:20.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:20.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:29:20.207 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:29:20.207 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:29:20.208 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:29:20.208 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:29:20.208 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:29:20.208 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:29:20.208 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:29:20.208 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:29:20.208 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:29:20.209 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:20.209 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:20.209 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:29:20.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:20.209 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:29:20.209 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:29:20.209 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:29:20.209 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:20.210 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:29:20.211 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:29:20.448 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~219s 2020-11-09T18:29:25.212 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:29:25.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:29:25.213 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:29:25.255 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:29:25.255 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:29:25.523 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:29:25.523 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:29:25.523 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:29:25.524 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:29:25.524 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:29:25.524 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:29:25.524 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:29:25.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:25.524 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:29:25.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:29:25.525 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:29:25.525 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:29:25.525 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:29:25.525 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:29:25.525 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:29:25.525 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:29:25.525 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:29:25.526 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:29:25.526 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:29:25.526 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:25.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:25.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:29:25.526 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:29:25.527 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:29:25.527 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:29:25.527 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:29:25.527 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:29:25.527 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:29:25.527 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:29:25.527 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:29:25.528 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:29:25.528 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:25.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:25.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:29:25.528 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:29:25.528 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:29:25.529 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:29:25.529 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:29:25.529 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:29:25.529 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:29:25.530 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:29:25.530 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:29:25.530 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:29:25.530 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:25.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:25.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:29:25.531 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:29:25.531 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:29:25.531 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:29:25.531 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:29:25.531 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:29:25.532 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:29:25.532 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:29:25.532 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:29:25.532 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:29:25.532 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:25.532 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:25.532 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:29:25.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:25.533 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:29:25.533 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:29:25.533 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:29:25.533 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:25.533 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:29:25.534 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:29:26.054 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~224s 2020-11-09T18:29:30.535 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:29:30.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:29:30.537 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:29:30.577 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:29:30.577 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:29:30.841 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:29:30.842 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:29:30.842 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:29:30.842 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:29:30.843 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:29:30.843 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:29:30.843 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:29:30.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:30.843 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:29:30.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:29:30.843 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:29:30.844 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:29:30.844 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:29:30.844 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:29:30.844 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:29:30.844 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:29:30.844 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:29:30.844 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:29:30.845 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:29:30.845 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:30.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:30.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:29:30.845 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:29:30.845 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:29:30.846 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:29:30.846 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:29:30.846 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:29:30.846 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:29:30.846 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:29:30.846 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:29:30.846 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:29:30.847 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:30.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:30.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:29:30.847 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:29:30.847 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:29:30.847 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:29:30.847 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:29:30.848 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:29:30.848 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:29:30.848 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:29:30.848 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:29:30.848 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:29:30.848 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:30.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:30.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:29:30.849 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:29:30.849 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:29:30.849 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:29:30.849 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:29:30.849 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:29:30.849 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:29:30.850 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:29:30.850 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:29:30.850 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:29:30.850 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:30.850 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:30.850 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:29:30.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:30.850 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:29:30.851 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:29:30.851 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:29:30.851 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:30.851 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:29:30.852 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:29:31.660 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~230s 2020-11-09T18:29:35.853 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:29:35.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:29:35.855 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:29:35.896 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:29:35.896 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:29:36.267 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:29:36.267 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:29:36.267 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:29:36.268 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:29:36.268 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:29:36.268 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:29:36.268 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:29:36.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:36.268 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:29:36.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:29:36.269 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:29:36.269 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:29:36.269 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:29:36.269 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:29:36.269 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:29:36.269 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:29:36.269 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:29:36.270 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:29:36.270 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:29:36.270 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:36.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:36.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:29:36.270 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:29:36.270 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:29:36.271 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:29:36.271 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:29:36.271 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:29:36.271 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:29:36.271 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:29:36.271 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:29:36.271 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:29:36.272 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:36.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:36.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:29:36.272 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:29:36.272 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:29:36.272 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:29:36.272 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:29:36.273 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:29:36.273 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:29:36.273 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:29:36.273 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:29:36.273 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:29:36.273 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:36.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:36.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:29:36.274 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:29:36.274 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:29:36.274 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:29:36.274 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:29:36.274 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:29:36.274 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:29:36.275 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:29:36.275 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:29:36.275 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:29:36.275 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:36.275 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:36.275 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:29:36.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:36.276 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:29:36.276 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:29:36.276 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:29:36.276 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:36.276 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:29:36.277 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:29:37.265 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~235s 2020-11-09T18:29:41.278 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:29:41.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:29:41.280 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:29:41.325 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:29:41.326 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:29:41.590 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:29:41.591 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:29:41.591 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:29:41.592 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:29:41.592 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:29:41.592 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:29:41.592 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:29:41.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:41.594 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:29:41.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:29:41.594 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:29:41.595 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:29:41.595 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:29:41.595 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:29:41.595 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:29:41.596 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:29:41.596 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:29:41.596 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:29:41.597 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:29:41.597 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:41.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:41.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:29:41.598 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:29:41.598 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:29:41.599 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:29:41.599 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:29:41.599 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:29:41.599 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:29:41.599 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:29:41.599 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:29:41.600 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:29:41.600 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:41.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:41.600 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:29:41.600 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:29:41.600 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:29:41.600 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:29:41.601 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:29:41.601 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:29:41.601 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:29:41.601 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:29:41.601 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:29:41.601 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:29:41.601 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:41.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:41.602 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:29:41.602 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:29:41.602 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:29:41.602 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:29:41.602 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:29:41.606 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:29:41.606 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:29:41.606 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:29:41.606 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:29:41.607 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:29:41.607 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:41.607 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:41.607 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:29:41.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:41.608 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:29:41.608 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:29:41.608 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:29:41.608 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:41.609 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:29:41.610 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:29:42.871 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~241s 2020-11-09T18:29:46.612 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:29:46.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:29:46.613 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:29:46.653 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:29:46.653 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:29:46.917 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:29:46.918 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:29:46.918 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:29:46.918 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:29:46.918 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:29:46.918 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:29:46.919 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:29:46.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:46.919 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:29:46.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:29:46.919 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:29:46.919 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:29:46.919 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:29:46.920 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:29:46.920 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:29:46.920 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:29:46.920 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:29:46.920 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:29:46.920 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:29:46.920 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:46.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:46.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:29:46.921 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:29:46.921 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:29:46.921 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:29:46.921 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:29:46.921 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:29:46.921 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:29:46.922 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:29:46.922 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:29:46.922 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:29:46.922 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:46.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:46.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:29:46.922 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:29:46.923 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:29:46.923 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:29:46.923 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:29:46.923 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:29:46.923 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:29:46.923 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:29:46.923 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:29:46.924 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:29:46.924 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:46.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:46.924 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:29:46.924 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:29:46.924 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:29:46.924 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:29:46.924 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:29:46.925 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:29:46.925 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:29:46.925 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:29:46.925 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:29:46.925 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:29:46.925 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:46.925 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:46.926 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:29:46.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:46.926 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:29:46.926 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:29:46.926 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:29:46.926 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:46.926 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:29:46.927 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:29:48.477 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~247s 2020-11-09T18:29:51.928 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:29:51.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:29:51.930 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:29:51.969 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:29:51.969 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:29:52.238 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:29:52.239 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:29:52.239 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:29:52.239 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:29:52.239 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:29:52.239 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:29:52.240 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:29:52.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:52.240 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:29:52.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:29:52.240 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:29:52.240 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:29:52.241 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:29:52.241 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:29:52.241 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:29:52.241 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:29:52.241 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:29:52.241 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:29:52.242 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:29:52.242 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:52.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:52.242 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:29:52.242 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:29:52.242 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:29:52.243 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:29:52.243 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:29:52.243 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:29:52.243 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:29:52.243 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:29:52.243 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:29:52.243 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:29:52.244 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:52.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:52.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:29:52.244 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:29:52.244 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:29:52.244 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:29:52.244 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:29:52.245 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:29:52.245 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:29:52.245 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:29:52.245 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:29:52.245 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:29:52.245 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:52.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:52.245 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:29:52.246 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:29:52.246 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:29:52.246 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:29:52.246 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:29:52.246 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:29:52.246 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:29:52.247 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:29:52.247 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:29:52.247 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:29:52.247 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:52.247 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:52.247 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:29:52.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:52.248 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:29:52.248 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:29:52.248 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:29:52.248 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:52.248 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:29:52.249 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:29:54.083 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~252s 2020-11-09T18:29:57.251 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:29:57.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:29:57.252 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:29:57.294 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:29:57.294 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:29:57.554 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:29:57.555 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:29:57.555 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:29:57.555 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:29:57.555 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:29:57.556 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:29:57.556 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:29:57.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:57.556 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:29:57.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:29:57.556 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:29:57.557 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:29:57.557 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:29:57.557 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:29:57.557 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:29:57.557 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:29:57.557 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:29:57.558 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:29:57.558 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:29:57.558 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:57.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:57.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:29:57.559 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:29:57.559 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:29:57.559 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:29:57.559 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:29:57.559 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:29:57.559 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:29:57.559 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:29:57.560 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:29:57.560 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:29:57.560 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:57.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:57.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:29:57.560 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:29:57.560 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:29:57.561 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:29:57.561 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:29:57.561 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:29:57.561 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:29:57.561 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:29:57.561 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:29:57.562 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:29:57.562 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:57.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:57.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:29:57.562 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:29:57.563 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:29:57.563 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:29:57.563 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:29:57.563 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:29:57.563 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:29:57.563 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:29:57.563 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:29:57.563 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:29:57.564 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:57.564 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:57.564 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:29:57.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:29:57.564 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:29:57.564 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:29:57.564 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:29:57.565 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:29:57.565 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:29:57.566 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:29:59.689 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~258s 2020-11-09T18:30:02.567 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:30:02.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:30:02.568 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:30:02.609 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:30:02.609 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:30:02.880 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:30:02.881 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:30:02.881 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:30:02.881 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:30:02.881 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:30:02.881 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:30:02.882 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:30:02.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:02.882 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:30:02.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:30:02.882 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:30:02.882 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:30:02.883 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:30:02.883 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:30:02.883 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:30:02.883 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:30:02.883 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:30:02.883 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:30:02.883 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:30:02.884 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:02.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:02.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:30:02.884 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:30:02.884 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:30:02.884 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:30:02.884 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:30:02.885 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:30:02.885 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:30:02.885 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:30:02.885 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:30:02.886 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:30:02.886 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:02.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:02.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:30:02.886 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:30:02.886 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:30:02.887 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:30:02.887 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:30:02.887 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:30:02.887 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:30:02.887 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:30:02.887 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:30:02.887 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:30:02.888 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:02.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:02.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:30:02.891 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:30:02.891 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:30:02.891 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:30:02.891 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:30:02.892 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:30:02.892 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:30:02.892 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:30:02.892 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:30:02.893 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:30:02.893 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:02.893 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:02.893 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:30:02.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:02.893 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:30:02.893 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:30:02.894 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:30:02.894 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:02.894 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:30:02.895 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:30:05.295 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~263s 2020-11-09T18:30:07.896 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:30:07.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:30:07.897 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:30:07.937 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:30:07.937 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:30:08.209 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:30:08.209 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:30:08.209 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:30:08.209 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:30:08.210 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:30:08.210 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:30:08.210 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:30:08.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:08.210 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:30:08.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:30:08.210 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:30:08.211 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:30:08.211 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:30:08.211 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:30:08.211 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:30:08.211 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:30:08.211 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:30:08.211 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:30:08.212 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:30:08.212 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:08.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:08.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:30:08.212 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:30:08.212 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:30:08.213 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:30:08.213 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:30:08.213 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:30:08.213 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:30:08.213 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:30:08.213 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:30:08.213 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:30:08.214 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:08.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:08.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:30:08.214 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:30:08.214 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:30:08.214 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:30:08.214 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:30:08.215 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:30:08.215 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:30:08.215 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:30:08.215 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:30:08.215 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:30:08.215 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:08.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:08.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:30:08.216 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:30:08.216 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:30:08.216 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:30:08.216 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:30:08.216 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:30:08.216 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:30:08.216 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:30:08.217 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:30:08.217 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:30:08.217 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:08.217 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:08.217 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:30:08.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:08.217 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:30:08.218 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:30:08.218 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:30:08.218 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:08.218 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:30:08.219 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:30:10.901 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~269s 2020-11-09T18:30:13.220 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:30:13.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:30:13.221 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:30:13.261 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:30:13.261 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:30:13.528 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:30:13.529 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:30:13.529 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:30:13.529 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:30:13.530 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:30:13.530 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:30:13.530 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:30:13.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:13.531 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:30:13.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:30:13.532 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:30:13.532 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:30:13.532 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:30:13.533 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:30:13.533 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:30:13.533 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:30:13.534 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:30:13.534 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:30:13.534 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:30:13.535 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:13.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:13.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:30:13.536 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:30:13.536 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:30:13.536 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:30:13.537 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:30:13.537 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:30:13.538 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:30:13.538 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:30:13.539 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:30:13.539 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:30:13.539 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:13.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:13.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:30:13.540 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:30:13.541 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:30:13.541 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:30:13.541 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:30:13.542 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:30:13.542 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:30:13.542 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:30:13.543 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:30:13.543 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:30:13.543 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:13.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:13.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:30:13.544 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:30:13.544 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:30:13.545 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:30:13.545 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:30:13.545 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:30:13.546 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:30:13.546 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:30:13.546 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:30:13.547 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:30:13.547 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:13.548 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:13.548 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:30:13.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:13.549 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:30:13.549 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:30:13.549 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:30:13.549 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:13.549 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:30:13.550 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:30:16.507 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~275s 2020-11-09T18:30:18.552 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:30:18.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:30:18.553 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:30:18.597 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:30:18.598 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:30:18.863 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:30:18.864 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:30:18.864 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:30:18.864 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:30:18.864 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:30:18.865 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:30:18.865 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:30:18.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:18.865 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:30:18.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:30:18.866 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:30:18.867 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:30:18.868 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:30:18.868 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:30:18.868 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:30:18.868 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:30:18.868 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:30:18.868 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:30:18.869 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:30:18.869 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:18.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:18.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:30:18.869 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:30:18.869 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:30:18.870 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:30:18.870 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:30:18.870 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:30:18.871 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:30:18.871 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:30:18.871 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:30:18.872 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:30:18.872 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:18.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:18.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:30:18.873 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:30:18.873 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:30:18.873 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:30:18.873 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:30:18.874 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:30:18.874 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:30:18.874 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:30:18.875 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:30:18.875 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:30:18.875 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:18.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:18.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:30:18.875 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:30:18.876 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:30:18.876 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:30:18.876 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:30:18.876 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:30:18.876 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:30:18.876 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:30:18.877 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:30:18.877 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:30:18.877 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:18.877 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:18.877 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:30:18.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:18.877 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:30:18.878 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:30:18.878 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:30:18.878 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:18.878 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:30:18.879 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:30:22.113 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~280s 2020-11-09T18:30:23.880 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:30:23.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:30:23.882 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:30:23.925 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:30:23.925 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:30:24.195 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:30:24.195 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:30:24.196 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:30:24.196 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:30:24.196 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:30:24.196 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:30:24.196 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:30:24.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:24.197 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:30:24.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:30:24.197 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:30:24.197 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:30:24.197 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:30:24.197 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:30:24.198 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:30:24.198 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:30:24.198 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:30:24.198 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:30:24.198 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:30:24.198 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:24.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:24.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:30:24.199 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:30:24.199 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:30:24.199 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:30:24.199 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:30:24.199 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:30:24.199 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:30:24.200 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:30:24.200 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:30:24.200 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:30:24.200 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:24.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:24.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:30:24.200 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:30:24.201 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:30:24.201 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:30:24.201 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:30:24.201 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:30:24.201 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:30:24.201 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:30:24.201 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:30:24.202 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:30:24.202 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:24.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:24.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:30:24.202 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:30:24.202 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:30:24.202 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:30:24.203 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:30:24.203 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:30:24.203 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:30:24.203 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:30:24.203 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:30:24.203 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:30:24.203 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:24.203 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:24.204 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:30:24.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:24.204 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:30:24.204 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:30:24.204 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:30:24.204 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:24.204 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:30:24.205 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:30:27.718 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~286s 2020-11-09T18:30:29.207 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:30:29.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:30:29.208 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:30:29.250 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:30:29.250 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:30:29.522 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:30:29.522 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:30:29.522 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:30:29.522 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:30:29.522 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:30:29.523 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:30:29.523 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:30:29.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:29.523 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:30:29.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:30:29.523 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:30:29.524 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:30:29.524 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:30:29.524 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:30:29.524 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:30:29.524 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:30:29.524 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:30:29.524 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:30:29.525 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:30:29.525 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:29.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:29.525 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:30:29.525 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:30:29.525 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:30:29.525 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:30:29.525 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:30:29.526 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:30:29.526 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:30:29.526 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:30:29.526 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:30:29.526 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:30:29.526 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:29.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:29.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:30:29.527 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:30:29.527 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:30:29.527 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:30:29.527 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:30:29.527 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:30:29.527 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:30:29.528 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:30:29.528 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:30:29.528 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:30:29.528 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:29.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:29.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:30:29.528 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:30:29.529 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:30:29.529 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:30:29.529 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:30:29.529 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:30:29.529 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:30:29.529 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:30:29.529 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:30:29.530 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:30:29.530 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:29.530 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:29.530 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:30:29.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:29.530 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:30:29.530 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:30:29.530 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:30:29.531 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:29.531 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:30:29.532 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:30:33.324 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~292s 2020-11-09T18:30:34.533 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:30:34.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:30:34.534 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:30:34.575 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:30:34.576 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:30:34.941 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:30:34.942 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:30:34.942 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:30:34.942 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:30:34.942 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:30:34.942 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:30:34.943 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:30:34.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:34.943 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:30:34.943 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:30:34.943 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:30:34.943 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:30:34.944 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:30:34.944 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:30:34.944 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:30:34.944 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:30:34.944 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:30:34.944 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:30:34.944 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:30:34.945 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:34.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:34.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:30:34.945 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:30:34.945 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:30:34.946 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:30:34.946 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:30:34.946 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:30:34.946 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:30:34.947 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:30:34.947 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:30:34.947 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:30:34.947 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:34.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:34.947 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:30:34.948 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:30:34.948 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:30:34.948 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:30:34.948 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:30:34.948 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:30:34.948 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:30:34.948 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:30:34.949 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:30:34.949 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:30:34.949 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:34.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:34.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:30:34.949 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:30:34.950 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:30:34.950 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:30:34.950 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:30:34.950 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:30:34.950 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:30:34.950 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:30:34.951 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:30:34.951 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:30:34.951 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:34.951 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:34.951 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:30:34.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:34.951 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:30:34.952 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:30:34.952 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:30:34.952 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:34.952 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:30:34.953 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:30:38.928 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~297s 2020-11-09T18:30:39.954 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:30:39.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:30:39.955 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:30:39.995 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:30:39.996 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:30:40.253 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:30:40.253 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:30:40.254 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:30:40.254 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:30:40.254 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:30:40.254 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:30:40.254 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:30:40.254 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:40.255 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:30:40.255 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:30:40.255 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:30:40.255 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:30:40.255 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:30:40.255 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:30:40.255 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:30:40.256 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:30:40.256 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:30:40.256 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:30:40.256 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:30:40.256 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:40.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:40.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:30:40.257 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:30:40.257 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:30:40.257 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:30:40.257 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:30:40.257 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:30:40.257 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:30:40.257 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:30:40.258 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:30:40.258 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:30:40.258 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:40.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:40.258 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:30:40.258 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:30:40.259 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:30:40.259 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:30:40.259 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:30:40.259 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:30:40.259 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:30:40.259 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:30:40.259 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:30:40.260 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:30:40.260 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:40.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:40.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:30:40.260 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:30:40.260 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:30:40.260 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:30:40.261 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:30:40.261 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:30:40.261 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:30:40.261 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:30:40.261 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:30:40.261 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:30:40.261 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:40.262 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:40.262 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:30:40.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:40.262 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:30:40.262 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:30:40.262 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:30:40.262 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:40.263 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:30:40.264 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:30:44.535 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~303s 2020-11-09T18:30:44.535 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2020-11-09T18:30:44.535 ERROR:tasks.daemonwatchdog.daemon_watchdog:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-dang-webdir-empty-object/qa/tasks/daemonwatchdog.py", line 37, in _run self.watch() File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-dang-webdir-empty-object/qa/tasks/daemonwatchdog.py", line 113, in watch self.bark() File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-dang-webdir-empty-object/qa/tasks/daemonwatchdog.py", line 53, in bark for mount in self.ctx.mounts.values(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/config.py", line 254, in __getattr__ raise AttributeError(name) AttributeError: mounts 2020-11-09T18:30:45.265 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:30:45.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:30:45.266 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:30:45.307 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:30:45.308 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:30:45.575 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:30:45.576 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:30:45.576 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:30:45.576 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:30:45.576 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:30:45.576 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:30:45.577 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:30:45.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:45.577 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:30:45.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:30:45.577 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:30:45.577 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:30:45.577 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:30:45.578 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:30:45.578 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:30:45.578 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:30:45.578 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:30:45.578 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:30:45.578 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:30:45.578 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:45.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:45.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:30:45.579 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:30:45.579 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:30:45.579 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:30:45.579 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:30:45.579 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:30:45.580 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:30:45.580 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:30:45.580 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:30:45.580 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:30:45.580 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:45.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:45.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:30:45.581 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:30:45.581 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:30:45.581 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:30:45.581 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:30:45.581 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:30:45.581 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:30:45.581 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:30:45.581 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:30:45.582 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:30:45.582 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:45.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:45.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:30:45.582 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:30:45.582 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:30:45.582 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:30:45.583 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:30:45.583 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:30:45.583 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:30:45.583 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:30:45.583 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:30:45.583 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:30:45.583 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:45.584 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:45.584 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:30:45.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:45.584 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:30:45.584 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:30:45.584 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:30:45.584 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:45.584 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:30:45.585 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:30:50.586 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:30:50.587 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:30:50.588 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:30:50.629 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:30:50.630 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:30:50.897 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:30:50.897 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:30:50.897 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:30:50.897 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:30:50.898 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:30:50.898 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:30:50.898 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:30:50.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:50.898 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:30:50.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:30:50.898 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:30:50.899 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:30:50.899 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:30:50.899 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:30:50.899 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:30:50.899 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:30:50.899 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:30:50.899 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:30:50.900 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:30:50.900 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:50.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:50.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:30:50.900 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:30:50.900 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:30:50.900 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:30:50.900 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:30:50.901 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:30:50.901 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:30:50.901 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:30:50.901 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:30:50.901 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:30:50.901 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:50.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:50.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:30:50.902 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:30:50.902 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:30:50.902 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:30:50.902 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:30:50.902 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:30:50.902 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:30:50.903 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:30:50.903 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:30:50.903 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:30:50.903 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:50.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:50.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:30:50.904 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:30:50.904 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:30:50.904 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:30:50.904 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:30:50.904 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:30:50.904 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:30:50.905 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:30:50.905 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:30:50.905 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:30:50.906 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:50.906 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:50.906 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:30:50.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:50.906 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:30:50.907 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:30:50.907 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:30:50.907 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:50.907 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:30:50.908 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:30:55.910 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:30:55.910 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:30:55.911 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:30:55.952 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:30:55.953 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:30:56.224 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:30:56.224 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:30:56.225 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:30:56.225 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:30:56.225 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:30:56.225 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:30:56.225 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:30:56.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:56.225 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:30:56.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:30:56.226 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:30:56.226 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:30:56.226 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:30:56.226 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:30:56.226 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:30:56.227 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:30:56.227 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:30:56.227 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:30:56.227 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:30:56.227 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:56.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:56.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:30:56.228 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:30:56.228 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:30:56.228 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:30:56.228 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:30:56.228 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:30:56.228 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:30:56.229 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:30:56.229 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:30:56.229 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:30:56.229 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:56.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:56.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:30:56.229 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:30:56.230 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:30:56.230 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:30:56.230 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:30:56.230 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:30:56.230 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:30:56.230 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:30:56.230 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:30:56.231 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:30:56.231 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:56.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:56.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:30:56.231 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:30:56.231 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:30:56.232 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:30:56.232 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:30:56.232 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:30:56.233 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:30:56.233 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:30:56.233 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:30:56.233 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:30:56.233 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:56.233 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:56.233 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:30:56.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:30:56.234 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:30:56.234 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:30:56.234 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:30:56.234 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:30:56.234 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:30:56.235 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:31:01.237 INFO:tasks.rgw_multisite_tests:test s3 records fetching ... FAIL 2020-11-09T18:31:01.238 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:31:01.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:31:01.239 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:31:01.279 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:31:01.279 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:31:01.529 INFO:teuthology.orchestra.run.smithi051.stderr:No --period given, using current period=f6258aba-300a-4d5d-ad93-a7c198a0e0a1 2020-11-09T18:31:01.562 INFO:teuthology.orchestra.run.smithi051.stdout:[ 2020-11-09T18:31:01.563 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:31:01.563 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604946340.774139_451.1", 2020-11-09T18:31:01.563 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "2020-11-09T18:25:40.774139Z" 2020-11-09T18:31:01.563 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:31:01.563 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:31:01.564 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:31:01.564 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:31:01.564 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:31:01.564 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:31:01.564 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:31:01.564 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:31:01.564 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:31:01.565 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:31:01.565 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:31:01.565 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:31:01.565 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:31:01.565 INFO:teuthology.orchestra.run.smithi051.stdout:] 2020-11-09T18:31:01.566 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-09T18:31:01.567 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:31:01.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:31:01.567 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:31:01.609 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:31:01.610 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:31:01.846 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:31:01.847 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:31:01.847 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:31:01.847 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:31:01.847 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:31:01.847 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:31:01.848 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:31:01.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:01.848 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:31:01.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:01.848 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:31:01.848 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:01.848 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:01.849 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946340.774139_451.1", 2020-11-09T18:31:01.849 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:01.849 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:31:01.849 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:01.849 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:40.774139Z", 2020-11-09T18:31:01.849 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:31:01.849 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:01.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:01.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:01.855 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:31:01.856 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:01.856 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:01.856 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:31:01.856 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:01.856 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:31:01.856 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:01.856 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:31:01.857 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:31:01.857 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:01.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:01.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:01.857 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:31:01.857 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:01.857 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:01.858 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:31:01.858 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:01.858 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:31:01.858 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:01.858 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:31:01.858 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:31:01.858 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:01.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:01.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:01.859 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:31:01.859 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:01.859 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:01.859 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:31:01.859 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:01.860 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:31:01.860 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:01.860 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:31:01.860 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:31:01.860 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:01.860 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:01.860 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:31:01.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:01.861 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:31:01.861 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:31:01.861 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:31:01.861 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:01.861 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:31:01.862 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-09T18:31:01.863 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-09T18:31:01.863 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:31:01.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:31:01.864 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:31:01.904 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:31:01.904 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:31:02.179 INFO:teuthology.orchestra.run.smithi051.stdout:{ 2020-11-09T18:31:02.179 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_status": { 2020-11-09T18:31:02.180 INFO:teuthology.orchestra.run.smithi051.stdout: "info": { 2020-11-09T18:31:02.180 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "sync", 2020-11-09T18:31:02.180 INFO:teuthology.orchestra.run.smithi051.stdout: "num_shards": 4, 2020-11-09T18:31:02.181 INFO:teuthology.orchestra.run.smithi051.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:31:02.181 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 4 2020-11-09T18:31:02.181 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:31:02.182 INFO:teuthology.orchestra.run.smithi051.stdout: "markers": [ 2020-11-09T18:31:02.182 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:31:02.182 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 0, 2020-11-09T18:31:02.182 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:31:02.182 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:31:02.182 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604946340.774139_451.1", 2020-11-09T18:31:02.182 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:31:02.183 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 1, 2020-11-09T18:31:02.183 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:31:02.183 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:25:40.774139Z", 2020-11-09T18:31:02.183 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 4 2020-11-09T18:31:02.183 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:31:02.183 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:31:02.183 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:31:02.184 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 1, 2020-11-09T18:31:02.184 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:31:02.184 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:31:02.184 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:31:02.184 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:31:02.184 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:31:02.184 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:31:02.185 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:31:02.185 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:31:02.185 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:31:02.185 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:31:02.185 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:31:02.185 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 2, 2020-11-09T18:31:02.185 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:31:02.185 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:31:02.186 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:31:02.186 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:31:02.186 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:31:02.186 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:31:02.186 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:31:02.186 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:31:02.186 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:31:02.187 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:31:02.187 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:31:02.187 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 3, 2020-11-09T18:31:02.187 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:31:02.187 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:31:02.188 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:31:02.188 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:31:02.188 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:31:02.188 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:31:02.188 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:31:02.188 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:31:02.189 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:31:02.189 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:31:02.189 INFO:teuthology.orchestra.run.smithi051.stdout: ] 2020-11-09T18:31:02.189 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:31:02.189 INFO:teuthology.orchestra.run.smithi051.stdout: "full_sync": { 2020-11-09T18:31:02.189 INFO:teuthology.orchestra.run.smithi051.stdout: "total": 1, 2020-11-09T18:31:02.189 INFO:teuthology.orchestra.run.smithi051.stdout: "complete": 1 2020-11-09T18:31:02.190 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:31:02.190 INFO:teuthology.orchestra.run.smithi051.stdout:} 2020-11-09T18:31:02.191 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-09T18:31:02.191 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-09T18:31:02.191 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:31:02.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:31:02.192 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:31:02.232 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:31:02.233 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:31:02.594 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:31:02.595 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:31:02.595 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:31:02.596 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:31:02.596 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:31:02.596 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:31:02.596 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:31:02.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:02.597 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:31:02.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:02.597 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:31:02.598 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:02.598 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:02.598 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:31:02.598 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:02.599 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:31:02.599 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:02.599 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:31:02.599 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:31:02.599 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:02.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:02.600 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:02.600 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:31:02.600 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:02.600 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:02.601 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:31:02.601 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:02.601 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:31:02.601 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:02.601 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:31:02.602 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:31:02.602 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:02.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:02.602 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:02.602 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:31:02.603 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:02.603 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:02.603 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:31:02.603 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:02.603 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:31:02.604 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:02.604 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:31:02.604 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:31:02.604 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:02.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:02.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:02.605 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:31:02.605 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:02.605 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:02.605 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:31:02.606 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:02.606 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:31:02.606 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:02.606 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:31:02.606 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:31:02.607 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:02.607 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:02.607 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:31:02.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:02.607 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:31:02.607 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:31:02.608 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:31:02.608 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:02.608 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:31:02.609 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:31:07.611 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:31:07.611 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:31:07.612 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:31:07.645 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:31:07.645 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:31:07.906 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:31:07.907 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:31:07.907 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:31:07.907 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:31:07.907 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:31:07.907 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:31:07.908 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:31:07.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:07.908 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:31:07.908 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:07.908 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:31:07.908 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:07.908 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:07.909 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:31:07.909 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:07.909 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:31:07.909 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:07.909 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:31:07.909 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:31:07.909 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:07.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:07.910 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:07.910 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:31:07.910 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:07.910 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:07.910 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:31:07.910 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:07.911 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:31:07.911 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:07.911 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:31:07.911 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:31:07.911 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:07.911 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:07.911 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:07.912 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:31:07.912 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:07.912 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:07.912 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:31:07.912 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:07.912 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:31:07.912 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:07.912 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:31:07.913 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:31:07.913 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:07.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:07.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:07.913 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:31:07.913 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:07.913 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:07.914 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:31:07.914 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:07.914 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:31:07.914 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:07.914 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:31:07.914 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:31:07.914 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:07.915 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:07.915 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:31:07.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:07.915 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:31:07.915 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:31:07.915 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:31:07.915 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:07.915 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:31:07.916 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:31:12.917 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:31:12.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:31:12.918 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:31:12.961 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:31:12.961 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:31:13.220 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:31:13.221 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:31:13.221 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:31:13.221 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:31:13.221 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:31:13.221 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:31:13.221 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:31:13.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:13.222 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:31:13.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:13.222 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:31:13.222 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:13.222 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:13.222 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:31:13.223 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:13.223 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:31:13.223 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:13.223 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:31:13.223 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:31:13.223 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:13.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:13.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:13.224 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:31:13.224 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:13.224 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:13.224 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:31:13.224 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:13.224 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:31:13.224 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:13.225 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:31:13.225 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:31:13.225 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:13.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:13.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:13.225 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:31:13.225 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:13.226 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:13.226 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:31:13.226 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:13.226 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:31:13.226 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:13.226 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:31:13.226 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:31:13.227 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:13.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:13.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:13.227 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:31:13.227 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:13.227 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:13.227 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:31:13.227 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:13.228 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:31:13.228 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:13.228 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:31:13.228 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:31:13.228 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:13.228 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:13.228 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:31:13.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:13.229 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:31:13.229 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:31:13.229 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:31:13.229 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:13.229 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:31:13.230 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:31:18.232 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:31:18.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:31:18.234 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:31:18.275 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:31:18.276 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:31:18.545 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:31:18.546 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:31:18.546 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:31:18.546 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:31:18.547 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:31:18.547 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:31:18.547 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:31:18.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:18.547 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:31:18.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:18.548 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:31:18.548 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:18.548 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:18.548 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:31:18.548 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:18.549 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:31:18.549 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:18.549 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:31:18.549 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:31:18.549 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:18.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:18.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:18.550 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:31:18.550 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:18.550 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:18.550 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:31:18.550 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:18.551 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:31:18.551 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:18.551 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:31:18.551 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:31:18.551 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:18.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:18.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:18.552 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:31:18.552 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:18.552 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:18.552 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:31:18.552 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:18.553 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:31:18.553 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:18.553 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:31:18.553 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:31:18.553 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:18.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:18.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:18.554 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:31:18.554 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:18.554 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:18.554 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:31:18.554 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:18.554 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:31:18.554 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:18.555 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:31:18.555 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:31:18.555 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:18.555 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:18.555 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:31:18.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:18.556 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:31:18.556 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:31:18.556 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:31:18.556 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:18.556 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:31:18.557 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:31:23.559 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:31:23.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:31:23.560 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:31:23.603 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:31:23.604 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:31:23.862 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:31:23.863 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:31:23.863 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:31:23.863 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:31:23.863 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:31:23.864 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:31:23.864 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:31:23.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:23.864 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:31:23.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:23.864 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:31:23.864 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:23.865 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:23.865 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:31:23.865 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:23.865 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:31:23.865 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:23.865 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:31:23.865 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:31:23.866 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:23.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:23.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:23.866 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:31:23.866 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:23.866 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:23.867 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:31:23.867 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:23.867 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:31:23.867 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:23.867 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:31:23.867 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:31:23.868 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:23.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:23.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:23.868 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:31:23.868 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:23.868 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:23.868 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:31:23.869 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:23.869 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:31:23.869 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:23.869 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:31:23.869 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:31:23.869 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:23.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:23.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:23.870 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:31:23.870 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:23.870 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:23.870 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:31:23.870 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:23.870 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:31:23.871 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:23.871 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:31:23.871 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:31:23.871 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:23.871 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:23.871 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:31:23.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:23.872 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:31:23.872 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:31:23.872 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:31:23.872 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:23.872 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:31:23.873 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:31:28.875 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:31:28.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:31:28.876 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:31:28.917 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:31:28.917 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:31:29.186 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:31:29.186 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:31:29.186 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:31:29.186 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:31:29.187 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:31:29.187 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:31:29.187 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:31:29.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:29.187 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:31:29.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:29.187 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:31:29.188 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:29.188 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:29.188 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:31:29.188 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:29.188 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:31:29.188 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:29.188 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:31:29.189 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:31:29.189 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:29.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:29.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:29.189 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:31:29.189 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:29.189 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:29.190 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:31:29.190 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:29.190 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:31:29.190 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:29.190 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:31:29.190 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:31:29.190 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:29.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:29.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:29.191 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:31:29.191 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:29.191 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:29.191 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:31:29.191 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:29.191 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:31:29.192 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:29.192 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:31:29.192 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:31:29.192 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:29.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:29.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:29.192 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:31:29.193 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:29.193 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:29.193 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:31:29.193 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:29.193 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:31:29.193 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:29.193 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:31:29.193 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:31:29.194 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:29.194 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:29.194 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:31:29.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:29.194 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:31:29.194 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:31:29.194 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:31:29.195 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:29.195 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:31:29.196 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:31:34.197 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:31:34.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:31:34.198 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:31:34.241 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:31:34.241 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:31:34.612 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:31:34.613 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:31:34.613 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:31:34.613 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:31:34.613 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:31:34.613 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:31:34.613 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:31:34.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:34.614 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:31:34.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:34.614 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:31:34.614 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:34.614 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:34.614 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:31:34.615 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:34.615 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:31:34.615 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:34.615 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:31:34.615 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:31:34.615 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:34.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:34.616 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:34.616 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:31:34.616 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:34.616 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:34.616 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:31:34.616 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:34.616 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:31:34.616 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:34.617 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:31:34.617 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:31:34.617 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:34.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:34.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:34.617 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:31:34.617 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:34.618 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:34.618 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:31:34.618 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:34.618 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:31:34.618 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:34.618 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:31:34.618 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:31:34.619 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:34.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:34.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:34.619 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:31:34.619 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:34.619 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:34.619 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:31:34.619 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:34.620 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:31:34.620 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:34.620 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:31:34.620 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:31:34.620 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:34.620 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:34.620 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:31:34.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:34.621 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:31:34.621 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:31:34.621 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:31:34.621 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:34.621 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:31:34.622 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:31:39.623 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:31:39.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:31:39.625 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:31:39.666 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:31:39.666 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:31:39.935 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:31:39.936 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:31:39.936 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:31:39.937 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:31:39.937 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:31:39.937 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:31:39.938 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:31:39.938 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:39.939 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:31:39.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:39.939 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:31:39.940 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:39.940 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:39.940 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:31:39.941 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:39.941 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:31:39.941 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:39.942 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:31:39.943 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:31:39.943 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:39.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:39.943 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:39.944 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:31:39.944 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:39.944 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:39.945 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:31:39.945 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:39.945 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:31:39.946 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:39.946 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:31:39.947 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:31:39.948 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:39.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:39.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:39.949 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:31:39.950 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:39.951 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:39.951 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:31:39.951 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:39.951 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:31:39.951 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:39.951 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:31:39.951 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:31:39.952 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:39.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:39.952 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:39.952 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:31:39.952 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:39.952 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:39.952 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:31:39.953 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:39.953 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:31:39.953 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:39.953 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:31:39.953 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:31:39.953 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:39.953 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:39.954 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:31:39.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:39.954 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:31:39.954 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:31:39.954 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:31:39.954 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:39.954 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:31:39.955 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:31:44.956 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:31:44.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:31:44.958 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:31:45.001 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:31:45.001 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:31:45.271 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:31:45.271 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:31:45.271 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:31:45.271 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:31:45.271 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:31:45.272 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:31:45.272 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:31:45.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:45.272 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:31:45.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:45.272 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:31:45.272 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:45.273 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:45.273 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:31:45.273 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:45.273 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:31:45.273 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:45.273 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:31:45.273 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:31:45.274 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:45.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:45.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:45.274 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:31:45.274 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:45.275 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:45.275 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:31:45.275 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:45.275 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:31:45.275 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:45.275 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:31:45.276 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:31:45.276 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:45.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:45.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:45.276 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:31:45.276 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:45.276 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:45.277 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:31:45.277 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:45.277 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:31:45.277 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:45.277 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:31:45.277 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:31:45.277 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:45.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:45.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:45.278 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:31:45.278 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:45.278 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:45.278 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:31:45.278 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:45.279 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:31:45.279 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:45.279 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:31:45.279 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:31:45.279 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:45.279 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:45.279 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:31:45.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:45.280 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:31:45.280 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:31:45.280 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:31:45.280 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:45.280 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:31:45.281 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:31:50.283 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:31:50.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:31:50.284 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:31:50.326 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:31:50.326 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:31:50.698 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:31:50.699 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:31:50.699 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:31:50.699 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:31:50.699 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:31:50.699 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:31:50.700 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:31:50.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:50.700 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:31:50.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:50.700 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:31:50.700 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:50.700 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:50.701 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:31:50.701 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:50.701 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:31:50.701 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:50.701 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:31:50.701 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:31:50.702 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:50.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:50.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:50.702 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:31:50.702 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:50.702 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:50.702 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:31:50.703 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:50.703 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:31:50.703 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:50.703 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:31:50.703 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:31:50.703 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:50.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:50.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:50.704 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:31:50.704 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:50.704 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:50.704 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:31:50.704 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:50.704 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:31:50.704 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:50.705 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:31:50.705 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:31:50.705 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:50.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:50.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:50.705 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:31:50.705 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:50.706 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:50.706 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:31:50.706 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:50.706 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:31:50.706 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:50.706 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:31:50.706 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:31:50.707 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:50.707 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:50.707 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:31:50.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:50.707 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:31:50.707 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:31:50.707 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:31:50.707 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:50.708 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:31:50.709 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:31:55.709 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:31:55.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:31:55.711 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:31:55.751 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:31:55.751 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:31:56.120 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:31:56.121 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:31:56.121 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:31:56.121 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:31:56.121 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:31:56.121 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:31:56.121 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:31:56.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:56.122 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:31:56.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:56.122 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:31:56.122 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:56.122 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:56.122 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:31:56.123 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:56.123 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:31:56.123 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:56.123 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:31:56.123 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:31:56.123 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:56.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:56.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:56.124 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:31:56.124 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:56.124 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:56.124 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:31:56.124 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:56.124 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:31:56.124 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:56.125 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:31:56.125 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:31:56.125 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:56.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:56.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:56.125 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:31:56.125 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:56.126 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:56.126 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:31:56.126 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:56.126 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:31:56.126 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:56.126 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:31:56.126 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:31:56.127 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:56.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:56.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:31:56.127 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:31:56.127 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:31:56.127 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:31:56.127 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:31:56.128 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:31:56.128 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:31:56.128 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:31:56.128 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:31:56.128 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:31:56.128 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:56.128 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:56.128 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:31:56.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:31:56.129 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:31:56.129 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:31:56.129 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:31:56.129 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:31:56.129 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:31:56.130 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:32:01.131 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:32:01.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:32:01.133 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:32:01.173 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:32:01.174 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:32:01.545 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:32:01.546 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:32:01.546 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:32:01.546 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:32:01.546 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:32:01.547 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:32:01.547 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:32:01.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:01.547 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:32:01.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:32:01.547 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:32:01.548 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:32:01.548 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:32:01.548 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:32:01.548 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:32:01.548 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:32:01.548 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:32:01.548 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:32:01.549 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:32:01.549 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:01.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:01.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:32:01.549 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:32:01.549 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:32:01.550 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:32:01.550 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:32:01.550 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:32:01.550 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:32:01.550 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:32:01.550 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:32:01.550 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:32:01.551 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:01.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:01.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:32:01.551 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:32:01.551 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:32:01.551 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:32:01.551 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:32:01.551 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:32:01.552 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:32:01.552 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:32:01.552 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:32:01.552 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:32:01.552 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:01.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:01.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:32:01.553 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:32:01.553 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:32:01.553 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:32:01.553 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:32:01.553 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:32:01.553 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:32:01.553 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:32:01.554 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:32:01.554 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:32:01.554 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:01.554 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:01.554 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:32:01.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:01.554 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:32:01.555 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:32:01.555 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:32:01.555 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:01.555 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:32:01.556 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:32:06.557 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:32:06.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:32:06.558 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:32:06.602 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:32:06.603 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:32:06.857 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:32:06.857 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:32:06.858 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:32:06.858 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:32:06.858 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:32:06.858 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:32:06.858 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:32:06.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:06.859 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:32:06.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:32:06.859 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:32:06.859 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:32:06.859 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:32:06.859 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:32:06.859 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:32:06.859 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:32:06.860 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:32:06.860 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:32:06.860 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:32:06.860 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:06.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:06.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:32:06.860 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:32:06.861 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:32:06.861 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:32:06.861 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:32:06.861 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:32:06.861 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:32:06.861 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:32:06.861 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:32:06.862 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:32:06.862 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:06.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:06.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:32:06.862 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:32:06.862 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:32:06.862 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:32:06.862 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:32:06.863 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:32:06.863 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:32:06.863 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:32:06.863 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:32:06.863 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:32:06.863 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:06.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:06.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:32:06.864 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:32:06.864 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:32:06.864 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:32:06.864 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:32:06.864 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:32:06.864 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:32:06.865 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:32:06.865 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:32:06.865 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:32:06.865 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:06.865 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:06.865 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:32:06.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:06.865 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:32:06.866 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:32:06.866 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:32:06.866 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:06.866 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:32:06.867 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:32:11.868 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:32:11.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:32:11.869 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:32:11.909 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:32:11.909 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:32:12.270 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:32:12.271 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:32:12.271 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:32:12.271 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:32:12.271 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:32:12.271 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:32:12.272 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:32:12.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:12.272 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:32:12.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:32:12.272 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:32:12.272 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:32:12.273 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:32:12.273 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:32:12.273 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:32:12.273 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:32:12.273 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:32:12.273 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:32:12.274 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:32:12.274 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:12.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:12.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:32:12.275 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:32:12.275 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:32:12.275 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:32:12.275 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:32:12.275 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:32:12.275 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:32:12.275 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:32:12.276 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:32:12.276 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:32:12.276 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:12.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:12.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:32:12.276 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:32:12.276 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:32:12.277 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:32:12.277 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:32:12.277 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:32:12.277 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:32:12.277 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:32:12.277 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:32:12.277 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:32:12.278 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:12.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:12.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:32:12.278 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:32:12.278 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:32:12.278 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:32:12.278 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:32:12.279 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:32:12.279 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:32:12.279 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:32:12.279 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:32:12.279 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:32:12.279 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:12.279 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:12.280 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:32:12.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:12.280 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:32:12.280 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:32:12.280 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:32:12.280 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:12.280 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:32:12.281 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:32:17.283 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:32:17.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:32:17.284 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:32:17.325 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:32:17.325 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:32:17.594 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:32:17.594 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:32:17.595 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:32:17.595 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:32:17.595 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:32:17.595 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:32:17.595 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:32:17.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:17.595 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:32:17.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:32:17.596 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:32:17.596 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:32:17.596 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:32:17.596 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:32:17.596 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:32:17.597 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:32:17.597 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:32:17.597 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:32:17.597 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:32:17.597 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:17.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:17.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:32:17.598 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:32:17.598 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:32:17.598 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:32:17.598 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:32:17.598 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:32:17.598 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:32:17.598 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:32:17.599 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:32:17.599 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:32:17.599 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:17.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:17.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:32:17.599 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:32:17.599 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:32:17.600 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:32:17.600 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:32:17.600 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:32:17.600 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:32:17.600 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:32:17.600 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:32:17.600 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:32:17.601 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:17.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:17.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:32:17.601 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:32:17.601 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:32:17.601 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:32:17.601 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:32:17.602 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:32:17.602 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:32:17.602 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:32:17.602 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:32:17.602 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:32:17.602 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:17.603 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:17.603 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:32:17.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:17.603 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:32:17.603 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:32:17.603 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:32:17.603 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:17.604 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:32:17.605 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:32:22.606 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:32:22.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:32:22.608 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:32:22.645 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:32:22.646 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:32:22.901 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:32:22.902 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:32:22.902 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:32:22.902 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:32:22.902 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:32:22.902 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:32:22.902 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:32:22.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:22.903 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:32:22.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:32:22.903 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:32:22.903 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:32:22.903 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:32:22.903 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:32:22.904 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:32:22.904 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:32:22.904 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:32:22.904 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:32:22.904 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:32:22.904 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:22.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:22.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:32:22.905 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:32:22.905 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:32:22.905 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:32:22.905 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:32:22.905 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:32:22.906 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:32:22.906 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:32:22.906 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:32:22.906 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:32:22.906 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:22.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:22.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:32:22.907 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:32:22.907 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:32:22.907 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:32:22.907 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:32:22.907 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:32:22.907 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:32:22.908 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:32:22.908 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:32:22.908 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:32:22.908 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:22.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:22.908 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:32:22.908 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:32:22.909 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:32:22.909 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:32:22.909 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:32:22.909 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:32:22.909 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:32:22.910 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:32:22.910 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:32:22.910 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:32:22.910 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:22.910 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:22.911 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:32:22.911 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:22.911 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:32:22.911 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:32:22.911 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:32:22.911 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:22.911 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:32:22.912 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:32:27.913 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:32:27.915 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:32:27.915 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:32:27.958 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:32:27.958 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:32:28.325 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:32:28.325 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:32:28.325 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:32:28.326 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:32:28.326 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:32:28.326 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:32:28.326 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:32:28.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:28.326 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:32:28.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:32:28.327 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:32:28.327 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:32:28.327 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:32:28.327 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:32:28.327 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:32:28.327 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:32:28.327 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:32:28.328 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:32:28.328 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:32:28.328 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:28.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:28.328 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:32:28.328 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:32:28.328 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:32:28.329 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:32:28.329 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:32:28.329 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:32:28.329 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:32:28.329 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:32:28.329 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:32:28.329 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:32:28.330 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:28.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:28.330 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:32:28.330 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:32:28.330 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:32:28.331 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:32:28.331 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:32:28.331 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:32:28.331 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:32:28.331 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:32:28.331 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:32:28.331 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:32:28.332 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:28.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:28.332 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:32:28.332 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:32:28.332 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:32:28.332 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:32:28.332 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:32:28.333 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:32:28.333 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:32:28.333 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:32:28.333 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:32:28.333 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:32:28.333 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:28.333 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:28.334 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:32:28.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:28.334 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:32:28.334 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:32:28.334 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:32:28.335 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:28.335 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:32:28.336 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:32:33.337 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:32:33.338 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:32:33.338 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:32:33.380 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:32:33.381 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:32:33.643 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:32:33.643 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:32:33.643 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:32:33.643 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:32:33.644 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:32:33.644 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:32:33.644 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:32:33.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:33.644 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:32:33.644 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:32:33.644 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:32:33.645 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:32:33.645 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:32:33.645 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:32:33.645 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:32:33.645 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:32:33.645 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:32:33.645 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:32:33.646 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:32:33.646 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:33.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:33.646 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:32:33.646 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:32:33.646 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:32:33.646 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:32:33.647 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:32:33.647 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:32:33.647 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:32:33.647 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:32:33.647 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:32:33.647 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:32:33.647 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:33.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:33.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:32:33.648 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:32:33.648 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:32:33.648 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:32:33.648 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:32:33.648 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:32:33.649 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:32:33.649 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:32:33.649 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:32:33.649 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:32:33.649 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:33.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:33.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:32:33.649 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:32:33.650 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:32:33.650 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:32:33.650 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:32:33.650 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:32:33.650 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:32:33.650 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:32:33.650 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:32:33.651 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:32:33.651 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:33.651 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:33.651 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:32:33.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:33.651 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:32:33.651 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:32:33.651 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:32:33.652 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:33.652 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:32:33.653 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:32:38.654 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:32:38.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:32:38.655 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:32:38.698 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:32:38.698 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:32:38.952 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:32:38.953 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:32:38.953 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:32:38.953 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:32:38.953 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:32:38.953 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:32:38.954 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:32:38.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:38.954 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:32:38.954 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:32:38.954 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:32:38.954 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:32:38.954 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:32:38.955 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:32:38.955 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:32:38.955 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:32:38.955 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:32:38.955 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:32:38.955 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:32:38.955 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:38.956 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:38.956 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:32:38.956 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:32:38.956 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:32:38.956 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:32:38.956 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:32:38.957 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:32:38.957 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:32:38.957 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:32:38.957 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:32:38.957 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:32:38.957 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:38.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:38.958 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:32:38.958 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:32:38.958 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:32:38.958 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:32:38.958 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:32:38.958 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:32:38.958 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:32:38.959 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:32:38.959 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:32:38.959 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:32:38.959 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:38.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:38.959 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:32:38.959 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:32:38.960 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:32:38.960 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:32:38.960 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:32:38.960 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:32:38.960 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:32:38.960 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:32:38.960 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:32:38.961 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:32:38.961 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:38.961 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:38.961 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:32:38.961 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:38.961 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:32:38.961 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:32:38.962 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:32:38.962 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:38.962 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:32:38.963 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:32:43.963 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:32:43.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:32:43.965 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:32:44.006 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:32:44.007 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:32:44.374 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:32:44.374 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:32:44.375 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:32:44.375 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:32:44.375 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:32:44.375 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:32:44.375 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:32:44.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:44.376 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:32:44.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:32:44.376 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:32:44.376 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:32:44.376 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:32:44.377 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:32:44.377 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:32:44.377 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:32:44.377 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:32:44.377 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:32:44.377 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:32:44.377 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:44.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:44.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:32:44.378 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:32:44.378 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:32:44.378 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:32:44.378 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:32:44.378 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:32:44.379 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:32:44.379 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:32:44.380 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:32:44.380 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:32:44.380 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:44.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:44.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:32:44.380 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:32:44.381 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:32:44.381 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:32:44.381 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:32:44.381 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:32:44.381 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:32:44.381 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:32:44.381 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:32:44.382 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:32:44.382 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:44.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:44.382 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:32:44.382 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:32:44.382 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:32:44.382 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:32:44.383 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:32:44.383 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:32:44.383 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:32:44.383 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:32:44.383 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:32:44.383 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:32:44.383 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:44.383 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:44.384 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:32:44.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:44.384 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:32:44.384 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:32:44.384 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:32:44.384 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:44.384 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:32:44.385 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:32:49.388 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:32:49.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:32:49.389 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:32:49.429 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:32:49.429 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:32:49.689 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:32:49.689 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:32:49.690 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:32:49.690 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:32:49.690 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:32:49.690 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:32:49.690 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:32:49.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:49.691 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:32:49.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:32:49.691 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:32:49.691 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:32:49.691 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:32:49.691 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:32:49.691 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:32:49.693 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:32:49.693 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:32:49.693 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:32:49.693 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:32:49.694 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:49.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:49.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:32:49.694 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:32:49.694 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:32:49.694 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:32:49.694 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:32:49.695 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:32:49.695 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:32:49.695 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:32:49.695 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:32:49.695 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:32:49.695 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:49.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:49.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:32:49.696 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:32:49.696 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:32:49.696 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:32:49.696 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:32:49.696 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:32:49.696 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:32:49.697 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:32:49.697 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:32:49.697 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:32:49.697 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:49.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:49.697 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:32:49.697 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:32:49.698 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:32:49.698 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:32:49.699 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:32:49.699 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:32:49.699 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:32:49.699 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:32:49.699 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:32:49.699 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:32:49.699 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:49.699 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:49.700 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:32:49.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:49.700 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:32:49.700 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:32:49.700 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:32:49.700 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:49.700 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:32:49.702 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:32:54.703 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:32:54.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:32:54.704 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:32:54.741 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:32:54.742 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:32:55.001 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:32:55.002 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:32:55.002 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:32:55.002 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:32:55.002 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:32:55.003 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:32:55.003 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:32:55.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:55.003 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:32:55.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:32:55.003 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:32:55.003 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:32:55.004 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:32:55.004 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:32:55.004 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:32:55.004 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:32:55.004 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:32:55.004 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:32:55.004 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:32:55.005 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:55.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:55.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:32:55.005 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:32:55.005 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:32:55.005 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:32:55.005 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:32:55.006 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:32:55.010 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:32:55.010 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:32:55.010 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:32:55.011 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:32:55.011 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:55.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:55.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:32:55.011 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:32:55.011 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:32:55.012 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:32:55.012 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:32:55.012 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:32:55.012 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:32:55.012 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:32:55.013 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:32:55.013 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:32:55.013 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:55.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:55.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:32:55.014 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:32:55.014 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:32:55.014 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:32:55.015 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:32:55.015 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:32:55.015 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:32:55.016 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:32:55.016 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:32:55.016 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:32:55.016 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:55.017 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:55.017 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:32:55.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:32:55.018 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:32:55.018 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:32:55.018 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:32:55.019 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:32:55.019 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:32:55.020 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:33:00.022 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:33:00.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:33:00.024 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:33:00.066 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:33:00.066 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:33:00.338 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:33:00.338 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:33:00.339 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:33:00.339 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:33:00.339 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:33:00.339 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:33:00.339 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:33:00.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:00.340 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:33:00.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:00.340 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:33:00.340 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:00.340 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:00.340 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:33:00.341 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:00.341 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:33:00.341 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:00.341 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:33:00.342 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:33:00.342 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:00.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:00.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:00.343 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:33:00.343 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:00.343 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:00.343 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:33:00.344 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:00.344 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:33:00.344 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:00.344 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:33:00.344 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:33:00.344 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:00.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:00.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:00.345 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:33:00.345 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:00.345 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:00.345 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:33:00.345 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:00.346 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:33:00.346 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:00.346 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:33:00.346 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:33:00.346 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:00.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:00.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:00.347 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:33:00.347 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:00.347 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:00.347 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:33:00.347 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:00.347 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:33:00.348 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:00.348 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:33:00.348 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:33:00.348 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:00.348 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:00.348 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:33:00.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:00.349 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:33:00.349 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:33:00.349 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:33:00.349 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:00.349 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:33:00.350 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:33:05.351 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:33:05.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:33:05.354 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:33:05.405 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:33:05.406 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:33:05.677 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:33:05.677 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:33:05.677 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:33:05.678 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:33:05.678 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:33:05.678 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:33:05.678 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:33:05.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:05.679 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:33:05.679 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:05.679 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:33:05.679 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:05.679 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:05.680 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:33:05.680 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:05.680 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:33:05.680 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:05.680 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:33:05.680 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:33:05.681 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:05.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:05.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:05.681 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:33:05.681 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:05.681 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:05.682 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:33:05.682 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:05.682 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:33:05.682 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:05.682 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:33:05.682 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:33:05.683 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:05.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:05.683 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:05.683 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:33:05.683 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:05.683 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:05.683 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:33:05.684 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:05.684 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:33:05.684 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:05.684 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:33:05.684 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:33:05.684 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:05.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:05.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:05.685 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:33:05.685 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:05.685 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:05.686 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:33:05.686 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:05.687 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:33:05.687 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:05.687 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:33:05.687 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:33:05.687 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:05.687 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:05.688 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:33:05.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:05.688 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:33:05.688 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:33:05.688 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:33:05.688 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:05.688 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:33:05.690 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:33:10.691 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:33:10.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:33:10.692 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:33:10.732 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:33:10.732 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:33:10.998 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:33:10.999 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:33:10.999 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:33:10.999 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:33:10.999 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:33:11.000 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:33:11.000 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:33:11.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:11.000 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:33:11.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:11.001 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:33:11.001 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:11.001 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:11.001 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:33:11.001 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:11.001 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:33:11.001 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:11.001 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:33:11.002 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:33:11.002 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:11.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:11.002 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:11.002 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:33:11.002 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:11.003 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:11.003 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:33:11.003 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:11.003 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:33:11.003 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:11.003 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:33:11.003 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:33:11.004 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:11.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:11.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:11.004 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:33:11.004 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:11.004 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:11.004 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:33:11.005 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:11.005 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:33:11.005 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:11.005 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:33:11.005 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:33:11.005 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:11.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:11.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:11.006 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:33:11.006 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:11.007 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:11.007 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:33:11.007 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:11.007 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:33:11.007 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:11.008 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:33:11.008 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:33:11.008 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:11.008 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:11.008 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:33:11.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:11.008 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:33:11.009 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:33:11.009 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:33:11.009 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:11.009 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:33:11.010 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:33:16.011 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:33:16.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:33:16.013 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:33:16.053 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:33:16.053 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:33:16.319 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:33:16.319 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:33:16.320 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:33:16.320 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:33:16.320 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:33:16.320 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:33:16.320 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:33:16.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:16.321 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:33:16.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:16.321 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:33:16.321 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:16.321 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:16.321 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:33:16.322 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:16.322 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:33:16.322 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:16.322 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:33:16.322 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:33:16.322 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:16.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:16.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:16.323 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:33:16.323 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:16.323 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:16.323 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:33:16.323 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:16.324 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:33:16.324 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:16.324 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:33:16.324 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:33:16.325 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:16.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:16.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:16.325 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:33:16.325 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:16.325 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:16.325 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:33:16.325 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:16.326 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:33:16.326 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:16.326 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:33:16.326 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:33:16.326 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:16.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:16.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:16.327 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:33:16.327 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:16.327 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:16.327 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:33:16.327 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:16.327 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:33:16.327 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:16.327 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:33:16.328 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:33:16.328 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:16.328 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:16.328 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:33:16.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:16.328 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:33:16.328 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:33:16.329 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:33:16.329 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:16.329 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:33:16.330 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:33:21.331 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:33:21.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:33:21.333 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:33:21.374 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:33:21.375 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:33:21.638 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:33:21.638 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:33:21.638 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:33:21.639 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:33:21.639 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:33:21.639 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:33:21.639 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:33:21.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:21.639 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:33:21.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:21.640 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:33:21.640 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:21.640 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:21.640 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:33:21.640 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:21.640 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:33:21.640 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:21.641 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:33:21.641 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:33:21.641 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:21.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:21.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:21.641 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:33:21.641 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:21.641 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:21.642 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:33:21.642 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:21.642 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:33:21.642 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:21.642 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:33:21.642 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:33:21.642 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:21.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:21.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:21.643 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:33:21.643 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:21.643 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:21.643 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:33:21.643 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:21.644 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:33:21.644 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:21.644 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:33:21.644 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:33:21.644 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:21.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:21.644 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:21.644 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:33:21.645 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:21.645 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:21.645 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:33:21.645 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:21.645 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:33:21.645 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:21.645 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:33:21.646 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:33:21.646 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:21.646 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:21.646 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:33:21.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:21.646 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:33:21.646 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:33:21.647 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:33:21.647 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:21.647 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:33:21.648 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:33:26.650 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:33:26.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:33:26.651 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:33:26.692 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:33:26.692 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:33:26.945 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:33:26.945 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:33:26.945 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:33:26.945 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:33:26.946 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:33:26.946 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:33:26.946 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:33:26.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:26.946 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:33:26.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:26.947 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:33:26.947 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:26.947 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:26.947 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:33:26.947 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:26.947 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:33:26.947 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:26.948 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:33:26.948 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:33:26.948 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:26.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:26.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:26.948 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:33:26.948 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:26.948 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:26.949 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:33:26.949 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:26.949 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:33:26.949 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:26.949 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:33:26.949 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:33:26.949 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:26.950 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:26.950 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:26.950 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:33:26.950 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:26.950 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:26.950 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:33:26.950 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:26.951 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:33:26.951 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:26.951 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:33:26.951 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:33:26.951 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:26.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:26.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:26.952 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:33:26.952 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:26.952 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:26.952 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:33:26.952 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:26.952 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:33:26.952 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:26.952 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:33:26.953 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:33:26.953 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:26.953 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:26.953 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:33:26.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:26.953 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:33:26.953 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:33:26.954 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:33:26.954 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:26.954 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:33:26.955 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:33:31.956 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:33:31.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:33:31.957 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:33:31.997 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:33:31.998 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:33:32.267 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:33:32.267 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:33:32.267 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:33:32.268 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:33:32.268 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:33:32.268 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:33:32.268 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:33:32.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:32.268 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:33:32.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:32.269 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:33:32.269 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:32.269 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:32.269 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:33:32.269 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:32.269 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:33:32.269 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:32.270 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:33:32.270 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:33:32.270 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:32.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:32.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:32.270 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:33:32.270 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:32.271 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:32.271 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:33:32.271 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:32.271 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:33:32.271 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:32.271 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:33:32.271 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:33:32.272 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:32.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:32.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:32.272 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:33:32.272 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:32.272 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:32.272 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:33:32.272 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:32.273 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:33:32.273 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:32.273 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:33:32.273 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:33:32.273 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:32.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:32.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:32.274 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:33:32.274 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:32.274 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:32.274 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:33:32.274 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:32.274 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:33:32.274 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:32.274 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:33:32.275 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:33:32.275 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:32.275 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:32.275 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:33:32.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:32.275 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:33:32.275 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:33:32.276 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:33:32.276 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:32.276 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:33:32.277 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:33:37.278 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:33:37.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:33:37.279 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:33:37.322 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:33:37.322 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:33:37.581 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:33:37.582 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:33:37.582 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:33:37.582 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:33:37.582 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:33:37.582 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:33:37.582 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:33:37.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:37.583 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:33:37.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:37.583 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:33:37.583 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:37.583 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:37.583 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:33:37.584 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:37.584 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:33:37.584 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:37.584 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:33:37.584 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:33:37.584 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:37.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:37.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:37.585 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:33:37.585 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:37.585 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:37.586 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:33:37.586 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:37.586 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:33:37.586 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:37.586 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:33:37.586 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:33:37.586 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:37.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:37.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:37.587 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:33:37.587 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:37.587 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:37.587 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:33:37.587 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:37.588 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:33:37.588 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:37.588 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:33:37.588 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:33:37.588 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:37.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:37.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:37.589 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:33:37.589 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:37.589 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:37.589 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:33:37.589 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:37.589 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:33:37.589 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:37.590 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:33:37.590 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:33:37.590 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:37.590 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:37.590 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:33:37.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:37.591 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:33:37.591 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:33:37.591 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:33:37.591 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:37.591 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:33:37.592 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:33:42.593 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:33:42.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:33:42.594 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:33:42.636 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:33:42.636 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:33:42.905 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:33:42.905 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:33:42.905 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:33:42.905 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:33:42.906 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:33:42.906 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:33:42.906 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:33:42.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:42.906 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:33:42.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:42.907 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:33:42.907 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:42.907 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:42.907 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:33:42.907 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:42.907 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:33:42.907 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:42.907 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:33:42.908 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:33:42.908 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:42.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:42.908 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:42.908 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:33:42.908 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:42.908 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:42.909 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:33:42.909 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:42.909 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:33:42.909 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:42.909 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:33:42.909 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:33:42.909 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:42.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:42.910 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:42.910 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:33:42.910 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:42.910 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:42.910 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:33:42.910 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:42.911 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:33:42.911 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:42.911 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:33:42.911 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:33:42.911 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:42.911 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:42.911 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:42.912 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:33:42.912 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:42.912 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:42.912 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:33:42.912 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:42.912 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:33:42.912 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:42.912 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:33:42.913 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:33:42.913 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:42.913 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:42.913 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:33:42.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:42.913 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:33:42.913 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:33:42.914 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:33:42.914 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:42.914 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:33:42.915 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:33:47.917 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:33:47.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:33:47.919 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:33:47.962 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:33:47.962 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:33:48.228 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:33:48.228 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:33:48.228 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:33:48.228 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:33:48.229 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:33:48.229 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:33:48.229 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:33:48.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:48.229 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:33:48.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:48.230 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:33:48.230 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:48.230 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:48.230 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:33:48.230 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:48.230 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:33:48.230 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:48.231 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:33:48.231 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:33:48.231 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:48.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:48.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:48.231 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:33:48.231 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:48.232 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:48.232 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:33:48.232 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:48.232 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:33:48.232 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:48.232 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:33:48.232 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:33:48.232 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:48.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:48.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:48.233 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:33:48.233 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:48.233 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:48.233 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:33:48.233 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:48.234 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:33:48.234 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:48.234 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:33:48.234 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:33:48.234 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:48.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:48.234 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:48.235 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:33:48.235 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:48.235 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:48.235 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:33:48.235 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:48.235 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:33:48.236 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:48.237 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:33:48.237 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:33:48.237 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:48.237 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:48.237 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:33:48.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:48.238 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:33:48.238 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:33:48.238 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:33:48.238 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:48.238 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:33:48.239 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:33:53.241 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:33:53.242 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:33:53.243 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:33:53.286 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:33:53.286 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:33:53.651 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:33:53.652 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:33:53.652 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:33:53.652 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:33:53.652 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:33:53.652 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:33:53.653 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:33:53.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:53.653 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:33:53.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:53.653 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:33:53.653 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:53.653 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:53.654 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:33:53.654 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:53.654 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:33:53.654 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:53.654 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:33:53.654 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:33:53.655 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:53.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:53.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:53.655 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:33:53.655 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:53.655 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:53.655 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:33:53.656 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:53.656 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:33:53.656 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:53.656 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:33:53.656 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:33:53.656 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:53.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:53.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:53.657 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:33:53.657 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:53.657 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:53.657 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:33:53.657 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:53.657 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:33:53.657 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:53.658 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:33:53.658 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:33:53.658 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:53.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:53.658 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:53.658 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:33:53.658 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:53.659 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:53.659 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:33:53.659 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:53.659 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:33:53.659 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:53.660 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:33:53.660 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:33:53.660 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:53.660 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:53.660 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:33:53.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:53.661 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:33:53.661 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:33:53.661 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:33:53.661 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:53.661 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:33:53.662 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:33:58.663 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:33:58.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:33:58.711 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:33:58.752 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:33:58.752 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:33:59.116 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:33:59.116 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:33:59.117 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:33:59.117 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:33:59.117 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:33:59.117 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:33:59.117 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:33:59.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:59.117 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:33:59.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:59.118 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:33:59.118 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:59.118 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:59.118 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:33:59.118 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:59.118 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:33:59.119 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:59.119 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:33:59.119 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:33:59.119 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:59.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:59.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:59.119 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:33:59.120 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:59.120 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:59.120 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:33:59.120 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:59.120 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:33:59.120 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:59.120 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:33:59.120 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:33:59.121 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:59.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:59.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:59.121 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:33:59.121 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:59.121 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:59.121 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:33:59.122 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:59.122 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:33:59.122 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:59.122 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:33:59.122 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:33:59.122 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:59.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:59.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:33:59.123 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:33:59.123 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:33:59.123 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:33:59.123 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:33:59.123 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:33:59.123 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:33:59.123 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:33:59.124 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:33:59.124 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:33:59.124 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:59.124 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:59.124 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:33:59.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:33:59.124 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:33:59.125 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:33:59.125 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:33:59.125 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:33:59.125 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:33:59.126 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:34:04.127 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:34:04.128 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:34:04.128 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:34:04.169 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:34:04.169 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:34:04.530 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:34:04.531 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:34:04.531 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:34:04.532 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:34:04.532 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:34:04.532 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:34:04.532 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:34:04.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:04.533 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:34:04.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:34:04.533 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:34:04.534 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:34:04.534 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:34:04.534 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:34:04.534 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:34:04.535 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:34:04.535 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:34:04.535 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:34:04.535 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:34:04.536 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:04.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:04.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:34:04.536 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:34:04.537 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:34:04.537 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:34:04.538 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:34:04.539 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:34:04.539 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:34:04.539 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:34:04.539 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:34:04.540 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:34:04.540 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:04.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:04.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:34:04.541 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:34:04.541 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:34:04.541 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:34:04.542 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:34:04.542 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:34:04.542 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:34:04.542 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:34:04.543 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:34:04.543 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:34:04.543 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:04.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:04.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:34:04.544 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:34:04.544 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:34:04.544 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:34:04.544 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:34:04.544 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:34:04.545 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:34:04.545 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:34:04.545 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:34:04.545 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:34:04.545 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:04.545 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:04.546 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:34:04.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:04.546 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:34:04.546 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:34:04.546 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:34:04.546 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:04.546 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:34:04.548 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:34:09.550 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:34:09.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:34:09.552 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:34:09.595 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:34:09.596 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:34:09.848 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:34:09.849 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:34:09.849 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:34:09.849 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:34:09.849 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:34:09.849 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:34:09.849 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:34:09.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:09.850 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:34:09.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:34:09.850 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:34:09.850 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:34:09.850 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:34:09.851 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:34:09.851 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:34:09.851 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:34:09.851 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:34:09.851 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:34:09.851 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:34:09.851 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:09.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:09.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:34:09.852 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:34:09.852 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:34:09.852 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:34:09.852 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:34:09.852 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:34:09.852 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:34:09.853 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:34:09.853 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:34:09.853 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:34:09.853 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:09.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:09.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:34:09.853 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:34:09.854 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:34:09.854 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:34:09.854 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:34:09.854 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:34:09.854 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:34:09.854 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:34:09.854 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:34:09.855 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:34:09.855 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:09.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:09.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:34:09.855 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:34:09.855 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:34:09.855 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:34:09.855 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:34:09.856 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:34:09.856 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:34:09.856 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:34:09.856 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:34:09.856 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:34:09.856 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:09.856 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:09.857 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:34:09.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:09.857 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:34:09.857 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:34:09.857 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:34:09.857 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:09.857 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:34:09.858 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:34:14.859 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:34:14.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:34:14.861 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:34:14.903 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:34:14.903 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:34:15.271 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:34:15.271 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:34:15.271 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:34:15.271 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:34:15.271 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:34:15.272 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:34:15.272 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:34:15.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:15.272 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:34:15.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:34:15.273 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:34:15.273 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:34:15.273 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:34:15.273 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:34:15.273 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:34:15.273 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:34:15.273 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:34:15.274 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:34:15.274 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:34:15.274 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:15.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:15.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:34:15.275 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:34:15.275 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:34:15.275 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:34:15.275 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:34:15.275 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:34:15.276 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:34:15.276 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:34:15.276 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:34:15.276 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:34:15.276 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:15.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:15.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:34:15.276 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:34:15.277 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:34:15.277 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:34:15.277 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:34:15.277 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:34:15.277 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:34:15.277 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:34:15.277 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:34:15.278 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:34:15.282 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:15.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:15.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:34:15.283 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:34:15.283 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:34:15.283 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:34:15.283 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:34:15.284 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:34:15.284 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:34:15.284 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:34:15.284 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:34:15.284 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:34:15.285 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:15.285 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:15.285 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:34:15.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:15.286 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:34:15.286 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:34:15.286 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:34:15.287 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:15.287 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:34:15.288 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:34:20.291 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:34:20.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:34:20.293 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:34:20.336 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:34:20.337 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:34:20.597 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:34:20.598 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:34:20.598 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:34:20.598 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:34:20.598 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:34:20.598 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:34:20.598 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:34:20.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:20.599 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:34:20.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:34:20.599 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:34:20.599 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:34:20.599 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:34:20.599 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:34:20.600 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:34:20.600 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:34:20.600 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:34:20.600 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:34:20.600 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:34:20.600 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:20.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:20.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:34:20.601 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:34:20.601 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:34:20.601 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:34:20.601 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:34:20.601 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:34:20.601 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:34:20.601 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:34:20.602 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:34:20.602 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:34:20.602 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:20.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:20.602 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:34:20.602 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:34:20.602 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:34:20.603 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:34:20.603 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:34:20.603 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:34:20.603 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:34:20.603 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:34:20.603 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:34:20.603 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:34:20.604 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:20.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:20.604 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:34:20.604 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:34:20.604 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:34:20.604 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:34:20.604 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:34:20.604 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:34:20.605 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:34:20.605 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:34:20.605 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:34:20.605 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:34:20.605 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:20.605 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:20.605 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:34:20.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:20.606 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:34:20.606 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:34:20.606 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:34:20.606 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:20.606 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:34:20.607 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:34:25.608 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:34:25.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:34:25.609 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:34:25.647 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:34:25.648 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:34:26.009 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:34:26.010 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:34:26.010 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:34:26.010 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:34:26.011 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:34:26.011 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:34:26.011 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:34:26.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:26.012 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:34:26.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:34:26.012 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:34:26.012 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:34:26.013 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:34:26.013 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:34:26.013 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:34:26.013 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:34:26.014 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:34:26.014 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:34:26.014 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:34:26.014 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:26.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:26.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:34:26.015 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:34:26.015 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:34:26.015 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:34:26.016 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:34:26.016 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:34:26.017 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:34:26.017 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:34:26.017 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:34:26.018 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:34:26.018 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:26.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:26.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:34:26.019 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:34:26.019 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:34:26.019 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:34:26.019 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:34:26.020 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:34:26.020 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:34:26.020 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:34:26.020 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:34:26.021 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:34:26.021 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:26.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:26.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:34:26.022 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:34:26.022 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:34:26.022 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:34:26.022 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:34:26.022 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:34:26.023 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:34:26.023 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:34:26.023 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:34:26.024 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:34:26.024 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:26.024 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:26.024 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:34:26.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:26.025 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:34:26.025 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:34:26.025 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:34:26.026 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:26.026 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:34:26.027 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:34:31.029 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:34:31.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:34:31.031 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:34:31.074 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:34:31.074 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:34:31.338 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:34:31.339 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:34:31.339 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:34:31.339 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:34:31.339 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:34:31.340 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:34:31.340 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:34:31.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:31.340 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:34:31.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:34:31.340 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:34:31.340 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:34:31.341 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:34:31.341 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:34:31.341 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:34:31.341 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:34:31.341 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:34:31.341 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:34:31.341 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:34:31.341 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:31.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:31.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:34:31.343 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:34:31.343 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:34:31.343 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:34:31.343 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:34:31.343 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:34:31.344 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:34:31.344 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:34:31.344 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:34:31.344 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:34:31.344 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:31.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:31.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:34:31.345 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:34:31.345 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:34:31.345 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:34:31.345 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:34:31.345 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:34:31.345 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:34:31.345 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:34:31.346 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:34:31.346 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:34:31.346 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:31.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:31.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:34:31.346 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:34:31.346 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:34:31.347 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:34:31.347 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:34:31.347 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:34:31.347 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:34:31.347 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:34:31.347 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:34:31.347 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:34:31.347 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:31.348 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:31.348 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:34:31.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:31.348 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:34:31.348 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:34:31.348 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:34:31.348 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:31.349 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:34:31.350 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:34:36.350 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:34:36.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:34:36.352 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:34:36.392 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:34:36.393 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:34:36.754 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:34:36.755 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:34:36.755 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:34:36.756 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:34:36.756 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:34:36.756 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:34:36.756 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:34:36.757 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:36.757 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:34:36.757 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:34:36.757 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:34:36.758 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:34:36.758 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:34:36.758 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:34:36.758 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:34:36.759 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:34:36.759 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:34:36.759 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:34:36.759 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:34:36.759 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:36.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:36.760 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:34:36.760 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:34:36.760 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:34:36.760 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:34:36.761 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:34:36.761 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:34:36.761 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:34:36.761 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:34:36.761 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:34:36.762 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:34:36.762 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:36.762 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:36.762 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:34:36.762 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:34:36.762 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:34:36.763 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:34:36.763 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:34:36.763 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:34:36.763 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:34:36.763 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:34:36.763 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:34:36.763 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:34:36.764 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:36.764 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:36.764 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:34:36.764 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:34:36.764 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:34:36.764 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:34:36.764 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:34:36.765 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:34:36.765 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:34:36.765 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:34:36.765 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:34:36.765 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:34:36.765 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:36.765 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:36.766 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:34:36.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:36.766 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:34:36.766 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:34:36.766 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:34:36.766 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:36.767 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:34:36.768 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:34:41.769 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:34:41.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:34:41.770 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:34:41.810 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:34:41.810 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:34:42.182 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:34:42.182 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:34:42.182 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:34:42.183 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:34:42.183 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:34:42.183 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:34:42.183 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:34:42.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:42.183 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:34:42.183 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:34:42.184 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:34:42.184 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:34:42.184 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:34:42.184 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:34:42.184 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:34:42.184 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:34:42.184 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:34:42.184 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:34:42.185 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:34:42.185 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:42.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:42.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:34:42.185 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:34:42.185 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:34:42.185 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:34:42.186 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:34:42.186 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:34:42.186 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:34:42.186 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:34:42.186 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:34:42.186 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:34:42.186 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:42.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:42.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:34:42.187 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:34:42.187 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:34:42.187 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:34:42.187 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:34:42.187 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:34:42.188 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:34:42.188 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:34:42.188 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:34:42.188 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:34:42.188 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:42.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:42.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:34:42.188 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:34:42.189 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:34:42.189 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:34:42.189 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:34:42.189 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:34:42.189 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:34:42.189 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:34:42.189 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:34:42.190 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:34:42.190 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:42.190 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:42.190 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:34:42.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:42.190 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:34:42.190 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:34:42.191 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:34:42.191 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:42.191 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:34:42.192 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:34:47.193 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:34:47.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:34:47.194 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:34:47.235 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:34:47.236 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:34:47.507 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:34:47.508 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:34:47.508 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:34:47.508 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:34:47.508 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:34:47.508 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:34:47.508 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:34:47.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:47.509 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:34:47.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:34:47.509 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:34:47.509 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:34:47.509 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:34:47.509 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:34:47.510 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:34:47.510 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:34:47.510 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:34:47.510 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:34:47.510 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:34:47.510 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:47.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:47.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:34:47.511 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:34:47.511 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:34:47.511 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:34:47.511 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:34:47.511 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:34:47.512 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:34:47.512 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:34:47.512 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:34:47.512 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:34:47.512 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:47.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:47.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:34:47.513 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:34:47.513 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:34:47.513 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:34:47.513 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:34:47.513 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:34:47.513 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:34:47.513 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:34:47.514 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:34:47.514 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:34:47.514 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:47.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:47.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:34:47.514 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:34:47.514 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:34:47.515 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:34:47.515 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:34:47.515 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:34:47.515 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:34:47.515 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:34:47.515 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:34:47.515 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:34:47.516 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:47.516 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:47.516 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:34:47.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:47.516 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:34:47.516 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:34:47.516 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:34:47.517 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:47.517 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:34:47.518 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:34:52.519 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:34:52.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:34:52.520 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:34:52.557 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:34:52.557 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:34:52.818 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:34:52.818 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:34:52.818 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:34:52.819 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:34:52.819 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:34:52.819 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:34:52.819 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:34:52.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:52.819 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:34:52.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:34:52.820 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:34:52.820 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:34:52.822 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:34:52.822 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:34:52.822 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:34:52.823 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:34:52.823 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:34:52.823 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:34:52.823 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:34:52.823 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:52.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:52.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:34:52.824 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:34:52.824 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:34:52.824 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:34:52.824 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:34:52.824 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:34:52.824 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:34:52.824 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:34:52.825 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:34:52.825 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:34:52.825 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:52.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:52.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:34:52.825 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:34:52.826 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:34:52.826 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:34:52.826 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:34:52.826 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:34:52.826 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:34:52.827 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:34:52.827 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:34:52.827 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:34:52.827 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:52.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:52.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:34:52.827 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:34:52.827 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:34:52.828 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:34:52.828 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:34:52.828 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:34:52.828 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:34:52.828 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:34:52.828 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:34:52.829 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:34:52.830 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:52.830 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:52.831 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:34:52.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:52.831 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:34:52.831 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:34:52.832 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:34:52.832 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:52.832 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:34:52.834 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:34:57.835 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:34:57.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:34:57.836 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:34:57.882 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:34:57.882 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:34:58.150 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:34:58.150 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:34:58.151 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:34:58.151 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:34:58.151 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:34:58.151 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:34:58.152 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:34:58.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:58.152 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:34:58.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:34:58.153 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:34:58.153 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:34:58.153 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:34:58.154 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:34:58.154 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:34:58.154 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:34:58.154 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:34:58.154 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:34:58.154 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:34:58.154 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:58.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:58.155 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:34:58.155 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:34:58.155 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:34:58.155 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:34:58.155 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:34:58.156 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:34:58.156 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:34:58.156 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:34:58.156 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:34:58.156 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:34:58.156 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:58.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:58.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:34:58.157 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:34:58.157 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:34:58.157 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:34:58.157 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:34:58.157 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:34:58.157 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:34:58.157 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:34:58.158 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:34:58.158 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:34:58.158 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:58.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:58.158 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:34:58.158 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:34:58.159 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:34:58.159 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:34:58.159 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:34:58.159 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:34:58.159 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:34:58.159 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:34:58.159 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:34:58.159 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:34:58.160 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:58.160 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:58.160 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:34:58.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:34:58.160 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:34:58.160 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:34:58.160 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:34:58.161 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:34:58.161 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:34:58.162 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:35:03.163 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:35:03.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:35:03.164 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:35:03.204 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:35:03.204 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:35:03.568 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:35:03.569 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:35:03.569 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:35:03.569 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:35:03.569 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:35:03.569 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:35:03.570 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:35:03.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:03.570 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:35:03.570 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:35:03.570 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:35:03.570 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:35:03.570 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:35:03.571 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:35:03.571 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:35:03.571 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:35:03.571 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:35:03.571 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:35:03.571 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:35:03.571 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:03.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:03.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:35:03.572 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:35:03.572 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:35:03.572 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:35:03.572 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:35:03.573 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:35:03.573 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:35:03.573 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:35:03.573 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:35:03.573 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:35:03.574 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:03.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:03.574 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:35:03.574 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:35:03.574 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:35:03.574 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:35:03.574 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:35:03.574 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:35:03.575 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:35:03.575 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:35:03.575 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:35:03.575 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:35:03.575 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:03.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:03.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:35:03.576 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:35:03.576 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:35:03.576 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:35:03.576 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:35:03.576 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:35:03.577 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:35:03.578 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:35:03.578 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:35:03.579 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:35:03.579 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:03.579 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:03.579 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:35:03.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:03.579 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:35:03.580 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:35:03.580 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:35:03.580 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:03.580 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:35:03.582 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:35:08.583 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:35:08.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:35:08.585 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:35:08.623 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:35:08.624 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:35:08.894 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:35:08.895 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:35:08.895 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:35:08.895 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:35:08.895 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:35:08.895 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:35:08.896 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:35:08.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:08.896 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:35:08.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:35:08.896 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:35:08.896 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:35:08.896 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:35:08.897 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:35:08.897 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:35:08.897 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:35:08.897 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:35:08.897 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:35:08.897 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:35:08.897 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:08.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:08.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:35:08.898 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:35:08.898 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:35:08.898 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:35:08.898 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:35:08.898 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:35:08.899 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:35:08.899 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:35:08.899 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:35:08.899 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:35:08.899 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:08.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:08.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:35:08.900 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:35:08.900 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:35:08.900 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:35:08.900 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:35:08.900 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:35:08.900 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:35:08.900 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:35:08.901 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:35:08.901 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:35:08.901 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:08.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:08.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:35:08.901 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:35:08.901 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:35:08.902 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:35:08.902 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:35:08.902 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:35:08.902 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:35:08.902 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:35:08.902 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:35:08.902 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:35:08.902 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:08.903 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:08.903 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:35:08.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:08.903 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:35:08.903 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:35:08.903 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:35:08.903 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:08.904 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:35:08.904 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:35:13.905 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:35:13.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:35:13.907 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:35:13.947 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:35:13.947 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:35:14.210 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:35:14.211 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:35:14.211 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:35:14.211 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:35:14.211 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:35:14.212 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:35:14.212 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:35:14.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:14.212 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:35:14.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:35:14.212 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:35:14.212 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:35:14.213 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:35:14.213 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:35:14.213 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:35:14.213 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:35:14.213 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:35:14.213 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:35:14.214 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:35:14.214 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:14.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:14.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:35:14.214 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:35:14.214 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:35:14.214 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:35:14.215 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:35:14.215 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:35:14.215 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:35:14.215 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:35:14.215 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:35:14.215 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:35:14.215 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:14.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:14.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:35:14.216 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:35:14.216 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:35:14.216 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:35:14.216 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:35:14.216 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:35:14.216 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:35:14.217 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:35:14.217 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:35:14.217 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:35:14.217 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:14.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:14.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:35:14.217 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:35:14.218 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:35:14.218 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:35:14.218 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:35:14.218 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:35:14.218 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:35:14.218 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:35:14.218 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:35:14.219 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:35:14.219 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:14.219 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:14.219 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:35:14.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:14.219 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:35:14.219 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:35:14.219 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:35:14.220 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:14.220 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:35:14.221 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:35:19.222 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:35:19.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:35:19.223 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:35:19.265 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:35:19.266 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:35:19.531 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:35:19.532 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:35:19.532 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:35:19.532 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:35:19.532 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:35:19.532 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:35:19.533 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:35:19.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:19.533 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:35:19.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:35:19.533 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:35:19.533 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:35:19.533 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:35:19.534 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:35:19.534 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:35:19.534 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:35:19.534 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:35:19.534 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:35:19.534 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:35:19.534 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:19.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:19.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:35:19.535 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:35:19.535 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:35:19.535 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:35:19.535 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:35:19.535 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:35:19.535 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:35:19.536 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:35:19.536 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:35:19.536 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:35:19.536 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:19.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:19.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:35:19.536 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:35:19.537 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:35:19.537 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:35:19.537 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:35:19.537 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:35:19.537 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:35:19.537 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:35:19.538 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:35:19.538 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:35:19.538 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:19.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:19.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:35:19.538 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:35:19.538 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:35:19.539 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:35:19.539 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:35:19.539 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:35:19.539 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:35:19.539 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:35:19.539 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:35:19.539 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:35:19.539 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:19.540 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:19.540 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:35:19.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:19.540 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:35:19.540 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:35:19.540 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:35:19.540 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:19.541 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:35:19.541 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:35:24.543 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:35:24.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:35:24.544 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:35:24.584 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:35:24.584 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:35:24.949 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:35:24.949 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:35:24.949 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:35:24.950 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:35:24.950 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:35:24.950 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:35:24.951 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:35:24.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:24.951 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:35:24.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:35:24.952 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:35:24.952 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:35:24.953 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:35:24.954 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:35:24.954 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:35:24.954 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:35:24.954 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:35:24.955 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:35:24.955 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:35:24.955 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:24.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:24.955 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:35:24.956 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:35:24.956 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:35:24.956 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:35:24.959 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:35:24.959 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:35:24.960 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:35:24.960 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:35:24.960 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:35:24.960 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:35:24.960 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:24.961 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:24.961 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:35:24.961 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:35:24.961 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:35:24.961 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:35:24.961 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:35:24.962 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:35:24.962 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:35:24.962 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:35:24.962 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:35:24.962 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:35:24.962 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:24.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:24.963 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:35:24.963 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:35:24.963 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:35:24.963 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:35:24.963 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:35:24.963 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:35:24.963 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:35:24.964 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:35:24.964 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:35:24.964 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:35:24.964 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:24.964 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:24.964 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:35:24.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:24.965 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:35:24.965 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:35:24.965 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:35:24.965 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:24.965 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:35:24.966 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:35:29.967 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:35:29.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:35:29.969 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:35:30.010 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:35:30.010 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:35:30.280 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:35:30.281 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:35:30.281 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:35:30.281 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:35:30.281 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:35:30.281 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:35:30.281 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:35:30.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:30.282 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:35:30.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:35:30.282 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:35:30.282 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:35:30.282 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:35:30.282 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:35:30.283 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:35:30.283 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:35:30.283 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:35:30.283 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:35:30.283 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:35:30.283 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:30.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:30.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:35:30.284 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:35:30.284 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:35:30.284 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:35:30.284 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:35:30.285 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:35:30.285 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:35:30.285 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:35:30.285 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:35:30.285 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:35:30.285 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:30.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:30.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:35:30.286 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:35:30.286 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:35:30.286 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:35:30.286 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:35:30.286 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:35:30.287 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:35:30.287 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:35:30.287 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:35:30.287 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:35:30.287 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:30.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:30.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:35:30.288 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:35:30.288 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:35:30.288 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:35:30.288 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:35:30.288 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:35:30.289 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:35:30.289 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:35:30.289 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:35:30.289 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:35:30.289 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:30.289 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:30.289 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:35:30.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:30.290 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:35:30.290 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:35:30.290 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:35:30.290 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:30.290 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:35:30.291 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:35:35.293 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:35:35.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:35:35.295 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:35:35.337 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:35:35.338 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:35:35.604 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:35:35.604 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:35:35.605 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:35:35.605 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:35:35.605 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:35:35.605 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:35:35.605 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:35:35.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:35.606 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:35:35.606 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:35:35.606 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:35:35.606 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:35:35.606 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:35:35.606 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:35:35.606 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:35:35.607 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:35:35.607 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:35:35.607 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:35:35.607 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:35:35.608 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:35.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:35.608 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:35:35.608 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:35:35.608 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:35:35.608 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:35:35.608 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:35:35.609 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:35:35.609 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:35:35.609 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:35:35.609 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:35:35.609 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:35:35.609 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:35.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:35.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:35:35.610 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:35:35.610 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:35:35.610 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:35:35.610 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:35:35.610 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:35:35.611 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:35:35.611 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:35:35.611 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:35:35.611 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:35:35.611 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:35.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:35.611 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:35:35.611 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:35:35.612 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:35:35.612 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:35:35.612 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:35:35.612 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:35:35.612 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:35:35.612 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:35:35.612 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:35:35.613 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:35:35.613 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:35.613 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:35.613 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:35:35.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:35.613 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:35:35.613 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:35:35.613 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:35:35.614 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:35.614 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:35:35.615 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:35:40.616 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:35:40.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:35:40.618 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:35:40.659 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:35:40.659 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:35:40.926 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:35:40.927 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:35:40.927 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:35:40.927 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:35:40.927 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:35:40.927 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:35:40.928 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:35:40.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:40.928 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:35:40.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:35:40.928 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:35:40.928 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:35:40.928 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:35:40.929 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:35:40.929 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:35:40.929 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:35:40.929 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:35:40.929 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:35:40.929 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:35:40.929 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:40.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:40.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:35:40.930 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:35:40.930 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:35:40.930 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:35:40.930 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:35:40.930 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:35:40.931 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:35:40.931 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:35:40.931 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:35:40.931 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:35:40.931 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:40.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:40.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:35:40.932 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:35:40.932 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:35:40.932 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:35:40.932 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:35:40.932 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:35:40.932 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:35:40.932 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:35:40.933 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:35:40.933 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:35:40.933 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:40.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:40.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:35:40.933 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:35:40.933 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:35:40.934 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:35:40.935 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:35:40.935 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:35:40.935 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:35:40.935 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:35:40.936 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:35:40.936 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:35:40.936 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:40.936 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:40.936 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:35:40.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:40.936 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:35:40.937 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:35:40.937 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:35:40.937 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:40.937 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:35:40.938 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:35:45.939 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:35:45.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:35:45.940 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:35:45.981 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:35:45.981 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:35:46.352 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:35:46.352 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:35:46.353 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:35:46.353 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:35:46.353 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:35:46.353 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:35:46.353 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:35:46.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:46.354 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:35:46.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:35:46.354 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:35:46.354 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:35:46.354 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:35:46.354 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:35:46.354 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:35:46.355 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:35:46.355 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:35:46.355 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:35:46.355 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:35:46.355 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:46.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:46.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:35:46.356 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:35:46.356 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:35:46.356 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:35:46.356 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:35:46.356 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:35:46.356 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:35:46.356 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:35:46.356 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:35:46.357 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:35:46.357 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:46.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:46.357 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:35:46.357 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:35:46.357 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:35:46.357 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:35:46.358 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:35:46.358 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:35:46.358 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:35:46.358 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:35:46.358 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:35:46.358 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:35:46.358 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:46.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:46.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:35:46.359 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:35:46.359 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:35:46.359 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:35:46.359 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:35:46.359 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:35:46.359 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:35:46.360 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:35:46.360 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:35:46.360 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:35:46.360 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:46.360 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:46.360 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:35:46.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:46.361 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:35:46.361 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:35:46.361 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:35:46.361 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:46.361 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:35:46.362 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:35:51.363 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:35:51.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:35:51.364 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:35:51.404 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:35:51.405 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:35:51.677 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:35:51.678 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:35:51.678 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:35:51.678 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:35:51.678 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:35:51.678 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:35:51.679 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:35:51.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:51.679 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:35:51.679 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:35:51.679 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:35:51.679 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:35:51.679 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:35:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:35:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:35:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:35:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:35:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:35:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:35:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:35:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:35:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:35:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:35:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:35:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:35:51.682 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:35:51.682 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:35:51.682 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:35:51.682 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:35:51.682 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:51.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:51.682 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:35:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:35:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:35:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:35:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:35:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:35:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:35:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:35:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:35:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:35:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:35:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:35:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:35:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:35:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:35:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:35:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:35:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:35:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:35:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:35:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:51.687 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:35:51.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:51.687 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:35:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:35:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:35:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:51.688 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:35:51.688 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:35:56.690 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:35:56.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:35:56.691 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:35:56.733 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:35:56.734 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:35:57.104 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:35:57.105 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:35:57.105 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:35:57.105 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:35:57.105 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:35:57.105 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:35:57.105 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:35:57.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:57.106 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:35:57.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:35:57.106 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:35:57.106 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:35:57.106 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:35:57.107 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:35:57.107 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:35:57.107 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:35:57.107 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:35:57.107 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:35:57.107 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:35:57.107 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:57.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:57.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:35:57.108 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:35:57.108 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:35:57.109 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:35:57.109 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:35:57.109 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:35:57.109 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:35:57.109 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:35:57.109 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:35:57.110 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:35:57.110 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:57.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:57.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:35:57.110 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:35:57.110 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:35:57.110 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:35:57.111 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:35:57.111 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:35:57.111 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:35:57.111 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:35:57.111 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:35:57.111 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:35:57.111 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:57.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:57.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:35:57.112 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:35:57.112 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:35:57.112 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:35:57.112 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:35:57.112 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:35:57.112 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:35:57.113 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:35:57.113 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:35:57.113 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:35:57.113 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:57.113 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:57.113 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:35:57.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:35:57.113 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:35:57.114 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:35:57.114 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:35:57.114 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:35:57.114 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:35:57.115 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:36:02.116 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:36:02.117 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:36:02.118 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:36:02.162 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:36:02.162 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:36:02.540 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:36:02.540 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:36:02.541 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:36:02.541 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:36:02.541 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:36:02.541 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:36:02.542 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:36:02.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:02.542 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:36:02.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:02.543 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:36:02.543 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:02.543 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:02.543 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:36:02.543 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:02.543 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:36:02.543 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:02.543 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:36:02.544 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:36:02.544 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:02.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:02.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:02.544 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:36:02.544 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:02.544 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:02.545 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:36:02.545 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:02.545 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:36:02.545 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:02.545 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:36:02.545 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:36:02.546 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:02.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:02.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:02.546 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:36:02.546 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:02.546 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:02.547 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:36:02.547 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:02.547 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:36:02.547 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:02.547 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:36:02.547 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:36:02.547 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:02.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:02.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:02.548 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:36:02.548 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:02.548 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:02.548 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:36:02.548 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:02.549 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:36:02.549 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:02.550 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:36:02.550 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:36:02.550 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:02.550 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:02.550 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:36:02.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:02.550 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:36:02.551 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:36:02.551 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:36:02.551 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:02.551 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:36:02.552 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:36:07.553 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:36:07.554 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:36:07.556 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:36:07.597 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:36:07.597 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:36:07.864 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:36:07.865 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:36:07.865 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:36:07.865 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:36:07.865 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:36:07.865 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:36:07.866 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:36:07.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:07.866 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:36:07.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:07.866 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:36:07.866 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:07.866 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:07.867 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:36:07.867 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:07.867 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:36:07.867 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:07.867 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:36:07.867 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:36:07.867 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:07.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:07.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:07.868 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:36:07.868 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:07.868 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:07.868 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:36:07.869 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:07.869 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:36:07.869 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:07.869 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:36:07.869 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:36:07.869 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:07.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:07.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:07.870 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:36:07.870 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:07.870 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:07.870 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:36:07.870 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:07.870 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:36:07.871 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:07.871 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:36:07.871 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:36:07.871 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:07.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:07.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:07.872 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:36:07.872 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:07.872 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:07.872 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:36:07.872 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:07.872 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:36:07.872 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:07.873 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:36:07.873 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:36:07.873 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:07.873 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:07.873 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:36:07.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:07.874 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:36:07.874 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:36:07.874 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:36:07.874 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:07.874 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:36:07.875 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:36:12.877 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:36:12.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:36:12.878 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:36:12.921 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:36:12.922 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:36:13.288 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:36:13.290 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:36:13.290 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:36:13.291 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:36:13.291 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:36:13.291 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:36:13.291 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:36:13.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:13.292 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:36:13.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:13.293 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:36:13.293 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:13.293 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:13.294 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:36:13.299 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:13.300 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:36:13.301 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:13.301 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:36:13.302 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:36:13.302 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:13.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:13.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:13.306 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:36:13.308 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:13.308 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:13.308 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:36:13.308 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:13.308 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:36:13.309 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:13.309 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:36:13.309 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:36:13.310 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:13.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:13.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:13.311 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:36:13.311 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:13.311 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:13.312 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:36:13.312 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:13.312 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:36:13.312 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:13.312 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:36:13.313 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:36:13.313 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:13.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:13.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:13.313 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:36:13.313 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:13.314 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:13.314 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:36:13.314 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:13.314 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:36:13.314 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:13.314 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:36:13.315 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:36:13.315 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:13.315 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:13.315 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:36:13.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:13.315 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:36:13.316 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:36:13.316 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:36:13.316 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:13.316 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:36:13.318 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:36:18.321 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:36:18.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:36:18.323 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:36:18.364 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:36:18.364 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:36:18.633 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:36:18.634 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:36:18.634 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:36:18.634 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:36:18.634 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:36:18.635 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:36:18.635 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:36:18.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:18.635 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:36:18.635 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:18.635 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:36:18.636 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:18.636 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:18.636 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:36:18.636 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:18.637 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:36:18.637 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:18.637 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:36:18.637 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:36:18.637 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:18.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:18.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:18.638 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:36:18.638 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:18.638 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:18.638 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:36:18.638 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:18.639 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:36:18.639 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:18.639 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:36:18.639 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:36:18.640 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:18.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:18.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:18.640 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:36:18.640 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:18.640 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:18.640 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:36:18.641 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:18.641 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:36:18.641 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:18.641 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:36:18.642 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:36:18.642 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:18.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:18.642 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:18.642 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:36:18.642 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:18.643 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:18.643 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:36:18.643 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:18.643 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:36:18.643 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:18.643 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:36:18.644 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:36:18.644 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:18.644 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:18.644 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:36:18.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:18.644 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:36:18.644 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:36:18.644 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:36:18.645 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:18.645 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:36:18.646 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:36:23.648 INFO:tasks.rgw_multisite_tests:test s3 notification set/get/delete ... FAIL 2020-11-09T18:36:23.649 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:36:23.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:36:23.651 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:36:23.717 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:36:23.717 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:36:23.934 INFO:teuthology.orchestra.run.smithi051.stderr:No --period given, using current period=f6258aba-300a-4d5d-ad93-a7c198a0e0a1 2020-11-09T18:36:23.970 INFO:teuthology.orchestra.run.smithi051.stdout:[ 2020-11-09T18:36:23.970 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:36:23.971 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604946340.774139_451.1", 2020-11-09T18:36:23.972 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "2020-11-09T18:25:40.774139Z" 2020-11-09T18:36:23.972 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:36:23.972 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:36:23.972 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:36:23.972 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:36:23.973 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:36:23.973 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:36:23.973 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:36:23.973 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:36:23.974 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:36:23.974 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:36:23.974 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:36:23.974 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:36:23.975 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:36:23.975 INFO:teuthology.orchestra.run.smithi051.stdout:] 2020-11-09T18:36:23.977 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-09T18:36:23.978 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:36:23.978 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:36:23.979 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:36:24.026 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:36:24.027 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:36:24.300 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:36:24.300 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:36:24.300 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:36:24.300 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:36:24.300 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:36:24.301 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:36:24.301 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:36:24.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:24.301 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:36:24.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:24.301 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:36:24.301 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:24.302 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:24.302 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946340.774139_451.1", 2020-11-09T18:36:24.302 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:24.302 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:36:24.302 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:24.302 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:40.774139Z", 2020-11-09T18:36:24.303 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:36:24.303 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:24.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:24.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:24.303 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:36:24.303 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:24.303 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:24.304 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:36:24.304 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:24.304 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:36:24.304 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:24.304 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:36:24.304 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:36:24.304 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:24.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:24.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:24.305 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:36:24.305 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:24.305 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:24.305 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:36:24.305 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:24.306 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:36:24.306 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:24.306 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:36:24.306 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:36:24.306 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:24.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:24.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:24.307 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:36:24.307 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:24.307 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:24.307 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:36:24.307 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:24.307 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:36:24.308 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:24.308 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:36:24.308 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:36:24.308 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:24.308 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:24.308 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:36:24.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:24.309 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:36:24.309 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:36:24.309 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:36:24.309 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:24.309 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:36:24.310 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-09T18:36:24.311 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-09T18:36:24.311 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:36:24.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:36:24.312 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:36:24.354 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:36:24.355 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:36:24.618 INFO:teuthology.orchestra.run.smithi051.stdout:{ 2020-11-09T18:36:24.619 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_status": { 2020-11-09T18:36:24.619 INFO:teuthology.orchestra.run.smithi051.stdout: "info": { 2020-11-09T18:36:24.619 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "sync", 2020-11-09T18:36:24.619 INFO:teuthology.orchestra.run.smithi051.stdout: "num_shards": 4, 2020-11-09T18:36:24.620 INFO:teuthology.orchestra.run.smithi051.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:36:24.620 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 4 2020-11-09T18:36:24.620 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:36:24.620 INFO:teuthology.orchestra.run.smithi051.stdout: "markers": [ 2020-11-09T18:36:24.620 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:36:24.620 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 0, 2020-11-09T18:36:24.620 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:36:24.621 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:36:24.621 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604946340.774139_451.1", 2020-11-09T18:36:24.621 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:36:24.621 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 1, 2020-11-09T18:36:24.621 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:36:24.621 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:25:40.774139Z", 2020-11-09T18:36:24.621 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 4 2020-11-09T18:36:24.622 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:36:24.622 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:36:24.622 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:36:24.622 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 1, 2020-11-09T18:36:24.622 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:36:24.622 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:36:24.622 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:36:24.623 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:36:24.623 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:36:24.623 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:36:24.623 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:36:24.623 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:36:24.623 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:36:24.623 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:36:24.624 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:36:24.624 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 2, 2020-11-09T18:36:24.624 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:36:24.624 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:36:24.624 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:36:24.624 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:36:24.624 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:36:24.624 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:36:24.625 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:36:24.625 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:36:24.625 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:36:24.625 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:36:24.625 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:36:24.625 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 3, 2020-11-09T18:36:24.625 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:36:24.626 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:36:24.626 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:36:24.626 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:36:24.626 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:36:24.626 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:36:24.626 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:36:24.626 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:36:24.627 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:36:24.627 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:36:24.627 INFO:teuthology.orchestra.run.smithi051.stdout: ] 2020-11-09T18:36:24.627 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:36:24.627 INFO:teuthology.orchestra.run.smithi051.stdout: "full_sync": { 2020-11-09T18:36:24.627 INFO:teuthology.orchestra.run.smithi051.stdout: "total": 1, 2020-11-09T18:36:24.627 INFO:teuthology.orchestra.run.smithi051.stdout: "complete": 1 2020-11-09T18:36:24.627 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:36:24.628 INFO:teuthology.orchestra.run.smithi051.stdout:} 2020-11-09T18:36:24.629 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-09T18:36:24.629 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-09T18:36:24.629 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:36:24.630 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:36:24.630 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:36:24.673 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:36:24.673 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:36:25.035 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:36:25.036 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:36:25.036 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:36:25.036 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:36:25.036 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:36:25.036 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:36:25.037 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:36:25.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:25.037 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:36:25.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:25.037 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:36:25.037 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:25.037 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:25.038 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:36:25.038 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:25.038 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:36:25.038 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:25.038 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:36:25.038 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:36:25.038 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:25.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:25.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:25.039 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:36:25.039 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:25.039 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:25.039 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:36:25.039 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:25.039 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:36:25.040 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:25.040 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:36:25.040 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:36:25.040 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:25.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:25.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:25.040 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:36:25.041 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:25.041 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:25.041 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:36:25.041 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:25.041 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:36:25.041 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:25.041 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:36:25.042 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:36:25.042 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:25.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:25.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:25.042 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:36:25.042 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:25.042 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:25.043 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:36:25.043 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:25.043 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:36:25.043 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:25.043 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:36:25.043 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:36:25.043 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:25.043 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:25.044 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:36:25.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:25.044 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:36:25.044 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:36:25.044 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:36:25.044 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:25.045 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:36:25.046 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:36:30.047 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:36:30.048 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:36:30.049 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:36:30.090 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:36:30.091 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:36:30.361 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:36:30.362 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:36:30.362 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:36:30.362 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:36:30.362 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:36:30.363 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:36:30.363 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:36:30.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:30.363 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:36:30.363 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:30.363 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:36:30.363 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:30.364 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:30.364 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:36:30.364 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:30.364 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:36:30.364 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:30.364 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:36:30.364 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:36:30.365 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:30.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:30.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:30.365 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:36:30.365 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:30.365 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:30.365 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:36:30.365 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:30.366 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:36:30.366 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:30.367 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:36:30.367 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:36:30.367 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:30.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:30.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:30.367 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:36:30.367 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:30.368 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:30.368 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:36:30.368 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:30.368 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:36:30.368 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:30.368 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:36:30.369 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:36:30.369 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:30.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:30.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:30.369 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:36:30.369 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:30.369 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:30.369 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:36:30.370 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:30.371 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:36:30.371 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:30.371 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:36:30.371 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:36:30.371 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:30.371 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:30.371 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:36:30.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:30.372 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:36:30.372 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:36:30.372 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:36:30.372 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:30.372 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:36:30.373 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:36:35.374 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:36:35.374 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:36:35.375 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:36:35.416 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:36:35.416 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:36:35.687 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:36:35.687 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:36:35.687 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:36:35.687 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:36:35.687 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:36:35.688 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:36:35.688 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:36:35.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:35.688 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:36:35.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:35.688 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:36:35.688 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:35.689 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:35.689 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:36:35.689 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:35.689 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:36:35.689 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:35.689 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:36:35.689 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:36:35.690 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:35.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:35.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:35.690 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:36:35.690 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:35.690 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:35.691 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:36:35.691 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:35.691 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:36:35.691 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:35.691 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:36:35.691 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:36:35.691 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:35.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:35.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:35.692 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:36:35.692 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:35.692 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:35.692 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:36:35.692 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:35.692 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:36:35.693 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:35.693 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:36:35.693 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:36:35.693 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:35.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:35.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:35.693 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:36:35.694 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:35.694 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:35.694 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:36:35.694 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:35.694 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:36:35.694 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:35.694 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:36:35.695 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:36:35.695 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:35.695 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:35.695 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:36:35.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:35.695 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:36:35.695 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:36:35.695 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:36:35.696 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:35.696 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:36:35.697 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:36:40.697 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:36:40.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:36:40.699 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:36:40.737 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:36:40.737 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:36:40.998 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:36:40.998 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:36:40.998 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:36:40.999 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:36:40.999 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:36:40.999 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:36:40.999 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:36:40.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:40.999 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:36:40.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:41.000 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:36:41.000 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:41.000 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:41.000 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:36:41.000 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:41.000 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:36:41.000 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:41.001 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:36:41.001 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:36:41.001 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:41.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:41.001 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:41.001 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:36:41.002 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:41.002 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:41.002 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:36:41.002 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:41.002 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:36:41.002 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:41.002 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:36:41.003 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:36:41.003 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:41.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:41.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:41.003 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:36:41.003 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:41.003 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:41.004 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:36:41.004 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:41.004 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:36:41.004 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:41.004 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:36:41.004 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:36:41.004 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:41.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:41.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:41.005 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:36:41.005 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:41.005 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:41.005 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:36:41.005 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:41.005 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:36:41.006 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:41.006 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:36:41.006 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:36:41.006 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:41.006 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:41.006 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:36:41.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:41.007 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:36:41.007 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:36:41.007 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:36:41.007 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:41.007 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:36:41.008 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:36:46.009 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:36:46.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:36:46.010 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:36:46.051 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:36:46.052 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:36:46.321 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:36:46.321 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:36:46.321 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:36:46.321 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:36:46.321 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:36:46.322 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:36:46.322 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:36:46.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:46.322 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:36:46.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:46.322 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:36:46.322 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:46.323 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:46.323 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:36:46.323 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:46.323 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:36:46.323 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:46.323 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:36:46.323 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:36:46.324 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:46.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:46.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:46.324 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:36:46.324 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:46.324 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:46.324 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:36:46.325 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:46.325 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:36:46.325 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:46.325 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:36:46.325 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:36:46.325 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:46.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:46.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:46.326 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:36:46.326 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:46.326 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:46.326 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:36:46.326 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:46.326 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:36:46.326 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:46.327 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:36:46.327 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:36:46.327 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:46.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:46.328 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:46.328 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:36:46.328 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:46.328 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:46.328 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:36:46.328 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:46.329 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:36:46.329 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:46.329 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:36:46.329 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:36:46.329 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:46.329 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:46.329 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:36:46.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:46.330 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:36:46.330 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:36:46.330 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:36:46.330 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:46.330 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:36:46.331 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:36:51.333 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:36:51.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:36:51.334 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:36:51.377 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:36:51.377 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:36:51.744 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:36:51.744 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:36:51.745 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:36:51.745 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:36:51.745 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:36:51.745 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:36:51.745 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:36:51.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:51.745 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:36:51.746 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:51.746 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:36:51.746 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:51.746 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:51.746 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:36:51.746 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:51.746 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:36:51.747 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:51.747 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:36:51.747 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:36:51.747 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:51.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:51.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:51.747 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:36:51.748 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:51.748 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:51.748 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:36:51.748 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:51.748 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:36:51.748 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:51.748 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:36:51.749 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:36:51.749 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:51.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:51.749 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:51.749 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:36:51.749 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:51.750 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:51.750 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:36:51.750 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:51.750 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:36:51.750 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:51.750 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:36:51.750 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:36:51.751 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:51.751 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:51.751 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:51.751 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:36:51.751 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:51.751 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:51.751 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:36:51.751 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:51.752 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:36:51.752 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:51.752 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:36:51.752 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:36:51.752 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:51.752 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:51.752 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:36:51.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:51.753 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:36:51.753 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:36:51.753 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:36:51.753 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:51.753 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:36:51.754 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:36:56.755 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:36:56.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:36:56.757 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:36:56.799 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:36:56.799 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:36:57.160 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:36:57.161 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:36:57.161 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:36:57.161 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:36:57.161 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:36:57.161 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:36:57.161 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:36:57.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:57.162 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:36:57.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:57.162 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:36:57.162 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:57.162 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:57.162 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:36:57.163 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:57.163 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:36:57.163 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:57.163 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:36:57.163 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:36:57.163 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:57.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:57.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:57.164 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:36:57.164 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:57.164 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:57.164 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:36:57.164 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:57.164 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:36:57.165 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:57.165 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:36:57.165 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:36:57.165 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:57.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:57.165 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:57.165 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:36:57.166 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:57.166 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:57.166 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:36:57.166 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:57.166 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:36:57.166 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:57.166 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:36:57.166 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:36:57.167 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:57.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:57.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:36:57.167 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:36:57.167 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:36:57.167 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:36:57.167 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:36:57.168 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:36:57.168 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:36:57.168 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:36:57.168 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:36:57.168 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:36:57.168 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:57.168 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:57.169 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:36:57.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:36:57.169 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:36:57.169 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:36:57.169 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:36:57.169 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:36:57.169 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:36:57.170 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:37:02.172 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:37:02.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:37:02.173 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:37:02.219 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:37:02.220 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:37:02.580 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:37:02.581 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:37:02.581 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:37:02.581 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:37:02.581 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:37:02.582 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:37:02.582 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:37:02.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:02.582 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:37:02.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:37:02.582 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:37:02.582 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:37:02.583 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:37:02.583 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:37:02.583 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:37:02.583 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:37:02.583 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:37:02.583 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:37:02.584 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:37:02.584 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:02.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:02.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:37:02.585 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:37:02.585 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:37:02.585 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:37:02.585 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:37:02.585 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:37:02.585 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:37:02.586 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:37:02.586 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:37:02.586 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:37:02.586 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:02.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:02.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:37:02.586 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:37:02.587 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:37:02.587 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:37:02.587 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:37:02.587 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:37:02.587 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:37:02.587 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:37:02.587 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:37:02.587 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:37:02.588 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:02.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:02.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:37:02.588 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:37:02.588 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:37:02.588 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:37:02.588 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:37:02.589 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:37:02.589 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:37:02.589 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:37:02.589 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:37:02.590 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:37:02.597 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:02.597 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:02.597 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:37:02.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:02.598 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:37:02.598 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:37:02.598 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:37:02.598 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:02.598 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:37:02.599 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:37:07.600 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:37:07.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:37:07.602 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:37:07.637 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:37:07.638 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:37:08.002 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:37:08.003 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:37:08.003 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:37:08.003 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:37:08.004 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:37:08.004 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:37:08.004 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:37:08.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:08.005 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:37:08.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:37:08.005 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:37:08.006 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:37:08.006 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:37:08.007 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:37:08.007 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:37:08.007 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:37:08.007 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:37:08.008 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:37:08.008 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:37:08.009 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:08.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:08.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:37:08.010 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:37:08.010 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:37:08.010 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:37:08.010 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:37:08.010 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:37:08.011 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:37:08.011 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:37:08.011 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:37:08.012 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:37:08.012 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:08.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:08.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:37:08.012 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:37:08.012 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:37:08.013 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:37:08.013 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:37:08.013 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:37:08.013 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:37:08.013 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:37:08.013 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:37:08.014 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:37:08.014 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:08.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:08.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:37:08.014 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:37:08.015 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:37:08.015 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:37:08.015 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:37:08.015 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:37:08.015 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:37:08.015 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:37:08.016 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:37:08.016 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:37:08.016 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:08.016 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:08.016 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:37:08.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:08.017 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:37:08.017 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:37:08.017 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:37:08.017 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:08.017 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:37:08.019 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:37:13.021 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:37:13.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:37:13.025 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:37:13.068 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:37:13.068 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:37:13.341 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:37:13.342 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:37:13.342 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:37:13.342 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:37:13.342 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:37:13.343 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:37:13.343 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:37:13.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:13.343 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:37:13.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:37:13.343 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:37:13.343 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:37:13.344 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:37:13.344 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:37:13.344 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:37:13.344 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:37:13.344 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:37:13.344 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:37:13.344 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:37:13.345 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:13.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:13.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:37:13.345 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:37:13.345 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:37:13.345 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:37:13.345 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:37:13.345 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:37:13.346 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:37:13.349 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:37:13.349 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:37:13.349 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:37:13.349 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:13.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:13.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:37:13.350 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:37:13.350 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:37:13.350 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:37:13.350 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:37:13.350 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:37:13.351 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:37:13.351 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:37:13.351 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:37:13.351 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:37:13.351 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:13.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:13.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:37:13.352 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:37:13.352 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:37:13.352 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:37:13.352 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:37:13.352 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:37:13.352 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:37:13.352 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:37:13.352 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:37:13.353 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:37:13.353 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:13.353 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:13.353 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:37:13.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:13.353 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:37:13.353 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:37:13.354 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:37:13.354 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:13.354 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:37:13.354 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:37:18.355 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:37:18.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:37:18.356 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:37:18.398 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:37:18.398 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:37:18.767 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:37:18.767 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:37:18.768 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:37:18.768 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:37:18.768 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:37:18.768 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:37:18.768 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:37:18.768 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:18.768 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:37:18.769 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:37:18.769 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:37:18.769 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:37:18.769 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:37:18.769 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:37:18.769 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:37:18.770 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:37:18.770 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:37:18.770 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:37:18.770 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:37:18.770 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:18.770 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:18.770 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:37:18.771 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:37:18.771 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:37:18.771 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:37:18.771 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:37:18.771 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:37:18.771 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:37:18.771 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:37:18.772 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:37:18.772 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:37:18.772 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:18.772 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:18.772 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:37:18.772 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:37:18.772 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:37:18.773 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:37:18.773 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:37:18.773 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:37:18.773 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:37:18.773 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:37:18.773 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:37:18.774 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:37:18.774 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:18.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:18.774 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:37:18.774 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:37:18.774 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:37:18.774 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:37:18.775 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:37:18.775 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:37:18.775 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:37:18.775 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:37:18.775 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:37:18.775 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:37:18.775 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:18.776 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:18.776 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:37:18.776 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:18.776 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:37:18.776 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:37:18.776 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:37:18.776 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:18.777 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:37:18.778 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:37:23.779 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:37:23.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:37:23.781 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:37:23.827 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:37:23.827 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:37:24.192 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:37:24.192 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:37:24.192 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:37:24.192 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:37:24.192 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:37:24.193 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:37:24.193 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:37:24.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:24.193 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:37:24.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:37:24.193 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:37:24.193 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:37:24.194 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:37:24.194 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:37:24.194 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:37:24.194 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:37:24.194 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:37:24.194 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:37:24.194 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:37:24.195 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:24.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:24.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:37:24.195 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:37:24.195 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:37:24.195 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:37:24.195 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:37:24.196 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:37:24.196 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:37:24.196 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:37:24.196 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:37:24.196 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:37:24.196 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:24.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:24.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:37:24.197 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:37:24.197 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:37:24.197 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:37:24.197 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:37:24.197 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:37:24.197 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:37:24.197 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:37:24.198 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:37:24.198 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:37:24.198 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:24.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:24.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:37:24.198 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:37:24.198 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:37:24.199 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:37:24.199 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:37:24.199 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:37:24.199 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:37:24.199 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:37:24.199 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:37:24.199 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:37:24.199 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:24.200 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:24.200 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:37:24.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:24.200 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:37:24.200 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:37:24.200 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:37:24.200 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:24.201 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:37:24.203 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:37:29.204 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:37:29.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:37:29.205 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:37:29.245 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:37:29.245 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:37:29.613 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:37:29.614 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:37:29.614 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:37:29.614 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:37:29.614 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:37:29.614 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:37:29.614 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:37:29.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:29.615 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:37:29.615 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:37:29.615 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:37:29.615 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:37:29.615 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:37:29.615 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:37:29.615 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:37:29.616 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:37:29.616 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:37:29.616 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:37:29.616 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:37:29.616 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:29.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:29.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:37:29.617 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:37:29.617 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:37:29.617 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:37:29.617 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:37:29.618 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:37:29.618 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:37:29.618 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:37:29.618 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:37:29.618 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:37:29.618 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:29.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:29.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:37:29.619 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:37:29.619 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:37:29.619 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:37:29.619 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:37:29.619 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:37:29.619 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:37:29.619 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:37:29.620 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:37:29.620 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:37:29.620 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:29.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:29.620 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:37:29.620 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:37:29.620 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:37:29.621 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:37:29.621 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:37:29.621 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:37:29.621 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:37:29.621 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:37:29.621 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:37:29.622 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:37:29.622 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:29.622 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:29.622 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:37:29.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:29.623 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:37:29.623 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:37:29.623 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:37:29.623 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:29.623 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:37:29.624 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:37:34.626 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:37:34.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:37:34.627 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:37:34.667 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:37:34.668 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:37:34.934 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:37:34.934 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:37:34.934 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:37:34.935 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:37:34.935 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:37:34.935 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:37:34.935 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:37:34.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:34.935 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:37:34.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:37:34.936 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:37:34.936 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:37:34.936 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:37:34.936 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:37:34.936 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:37:34.936 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:37:34.937 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:37:34.937 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:37:34.937 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:37:34.937 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:34.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:34.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:37:34.937 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:37:34.938 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:37:34.938 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:37:34.938 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:37:34.938 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:37:34.938 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:37:34.938 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:37:34.938 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:37:34.939 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:37:34.939 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:34.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:34.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:37:34.939 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:37:34.939 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:37:34.939 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:37:34.940 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:37:34.940 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:37:34.940 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:37:34.940 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:37:34.940 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:37:34.940 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:37:34.940 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:34.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:34.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:37:34.941 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:37:34.941 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:37:34.941 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:37:34.941 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:37:34.941 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:37:34.941 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:37:34.942 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:37:34.942 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:37:34.942 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:37:34.942 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:34.942 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:34.942 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:37:34.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:34.943 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:37:34.943 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:37:34.943 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:37:34.946 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:34.946 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:37:34.947 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:37:39.949 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:37:39.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:37:39.950 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:37:39.991 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:37:39.991 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:37:40.358 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:37:40.358 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:37:40.359 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:37:40.359 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:37:40.359 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:37:40.359 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:37:40.359 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:37:40.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:40.360 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:37:40.360 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:37:40.360 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:37:40.360 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:37:40.360 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:37:40.360 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:37:40.360 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:37:40.361 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:37:40.361 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:37:40.361 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:37:40.361 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:37:40.361 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:40.362 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:40.362 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:37:40.362 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:37:40.362 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:37:40.362 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:37:40.362 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:37:40.362 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:37:40.363 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:37:40.363 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:37:40.363 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:37:40.363 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:37:40.363 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:40.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:40.363 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:37:40.363 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:37:40.364 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:37:40.364 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:37:40.364 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:37:40.364 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:37:40.364 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:37:40.364 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:37:40.364 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:37:40.365 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:37:40.365 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:40.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:40.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:37:40.365 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:37:40.365 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:37:40.366 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:37:40.366 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:37:40.366 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:37:40.367 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:37:40.367 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:37:40.367 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:37:40.367 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:37:40.367 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:40.367 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:40.367 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:37:40.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:40.368 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:37:40.368 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:37:40.368 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:37:40.368 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:40.368 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:37:40.369 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:37:45.371 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:37:45.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:37:45.372 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:37:45.414 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:37:45.414 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:37:45.787 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:37:45.787 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:37:45.787 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:37:45.787 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:37:45.788 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:37:45.788 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:37:45.788 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:37:45.788 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:45.788 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:37:45.788 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:37:45.788 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:37:45.789 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:37:45.789 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:37:45.789 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:37:45.789 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:37:45.789 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:37:45.789 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:37:45.789 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:37:45.790 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:37:45.790 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:45.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:45.790 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:37:45.790 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:37:45.790 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:37:45.790 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:37:45.791 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:37:45.791 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:37:45.791 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:37:45.791 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:37:45.791 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:37:45.791 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:37:45.791 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:45.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:45.792 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:37:45.792 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:37:45.792 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:37:45.792 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:37:45.792 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:37:45.792 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:37:45.792 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:37:45.793 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:37:45.793 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:37:45.793 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:37:45.793 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:45.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:45.793 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:37:45.793 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:37:45.794 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:37:45.794 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:37:45.794 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:37:45.794 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:37:45.794 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:37:45.794 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:37:45.794 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:37:45.795 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:37:45.795 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:45.795 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:45.795 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:37:45.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:45.795 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:37:45.795 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:37:45.795 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:37:45.796 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:45.796 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:37:45.797 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:37:50.798 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:37:50.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:37:50.799 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:37:50.841 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:37:50.841 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:37:51.110 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:37:51.111 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:37:51.111 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:37:51.111 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:37:51.111 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:37:51.111 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:37:51.111 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:37:51.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:51.112 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:37:51.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:37:51.112 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:37:51.112 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:37:51.112 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:37:51.112 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:37:51.112 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:37:51.113 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:37:51.113 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:37:51.113 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:37:51.113 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:37:51.113 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:51.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:51.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:37:51.114 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:37:51.114 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:37:51.114 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:37:51.114 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:37:51.114 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:37:51.114 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:37:51.115 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:37:51.115 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:37:51.115 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:37:51.115 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:51.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:51.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:37:51.115 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:37:51.115 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:37:51.116 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:37:51.116 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:37:51.116 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:37:51.116 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:37:51.116 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:37:51.116 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:37:51.116 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:37:51.117 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:51.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:51.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:37:51.117 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:37:51.117 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:37:51.117 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:37:51.118 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:37:51.118 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:37:51.118 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:37:51.118 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:37:51.118 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:37:51.118 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:37:51.119 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:51.119 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:51.119 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:37:51.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:51.119 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:37:51.119 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:37:51.119 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:37:51.120 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:51.120 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:37:51.121 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:37:56.122 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:37:56.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:37:56.124 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:37:56.164 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:37:56.165 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:37:56.536 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:37:56.537 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:37:56.537 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:37:56.537 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:37:56.537 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:37:56.538 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:37:56.538 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:37:56.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:56.538 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:37:56.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:37:56.539 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:37:56.539 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:37:56.539 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:37:56.539 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:37:56.539 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:37:56.540 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:37:56.540 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:37:56.540 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:37:56.540 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:37:56.540 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:56.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:56.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:37:56.541 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:37:56.541 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:37:56.541 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:37:56.542 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:37:56.542 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:37:56.542 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:37:56.542 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:37:56.543 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:37:56.543 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:37:56.543 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:56.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:56.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:37:56.543 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:37:56.544 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:37:56.544 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:37:56.544 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:37:56.544 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:37:56.544 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:37:56.545 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:37:56.545 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:37:56.545 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:37:56.545 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:56.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:56.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:37:56.546 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:37:56.546 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:37:56.546 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:37:56.546 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:37:56.546 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:37:56.546 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:37:56.547 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:37:56.547 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:37:56.547 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:37:56.547 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:56.547 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:56.547 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:37:56.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:37:56.548 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:37:56.548 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:37:56.548 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:37:56.548 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:37:56.548 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:37:56.549 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:38:01.550 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:38:01.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:38:01.552 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:38:01.592 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:38:01.592 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:38:01.954 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:38:01.955 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:38:01.955 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:38:01.955 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:38:01.955 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:38:01.955 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:38:01.955 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:38:01.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:01.956 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:38:01.956 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:38:01.956 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:38:01.956 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:38:01.956 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:38:01.956 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:38:01.956 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:38:01.957 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:38:01.957 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:38:01.957 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:38:01.957 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:38:01.957 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:01.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:01.957 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:38:01.958 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:38:01.958 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:38:01.958 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:38:01.958 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:38:01.958 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:38:01.959 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:38:01.959 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:38:01.959 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:38:01.959 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:38:01.959 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:01.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:01.959 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:38:01.960 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:38:01.960 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:38:01.960 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:38:01.960 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:38:01.960 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:38:01.960 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:38:01.960 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:38:01.961 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:38:01.961 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:38:01.961 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:01.961 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:01.961 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:38:01.961 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:38:01.961 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:38:01.962 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:38:01.962 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:38:01.962 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:38:01.962 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:38:01.962 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:38:01.962 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:38:01.962 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:38:01.962 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:01.963 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:01.963 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:38:01.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:01.963 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:38:01.963 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:38:01.963 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:38:01.964 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:01.964 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:38:01.965 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:38:06.967 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:38:06.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:38:06.968 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:38:07.012 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:38:07.013 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:38:07.380 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:38:07.381 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:38:07.381 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:38:07.381 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:38:07.381 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:38:07.382 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:38:07.382 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:38:07.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:07.382 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:38:07.382 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:38:07.382 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:38:07.383 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:38:07.383 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:38:07.383 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:38:07.383 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:38:07.383 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:38:07.383 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:38:07.384 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:38:07.384 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:38:07.384 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:07.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:07.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:38:07.384 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:38:07.385 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:38:07.385 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:38:07.385 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:38:07.385 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:38:07.385 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:38:07.385 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:38:07.386 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:38:07.386 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:38:07.386 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:07.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:07.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:38:07.386 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:38:07.386 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:38:07.387 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:38:07.387 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:38:07.387 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:38:07.387 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:38:07.387 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:38:07.387 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:38:07.388 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:38:07.388 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:07.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:07.388 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:38:07.388 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:38:07.388 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:38:07.389 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:38:07.389 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:38:07.389 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:38:07.389 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:38:07.389 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:38:07.389 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:38:07.389 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:38:07.390 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:07.390 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:07.390 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:38:07.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:07.390 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:38:07.390 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:38:07.391 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:38:07.391 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:07.391 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:38:07.392 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:38:12.394 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:38:12.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:38:12.395 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:38:12.435 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:38:12.436 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:38:12.704 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:38:12.705 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:38:12.705 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:38:12.705 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:38:12.705 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:38:12.705 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:38:12.706 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:38:12.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:12.706 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:38:12.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:38:12.706 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:38:12.706 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:38:12.707 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:38:12.707 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:38:12.707 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:38:12.707 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:38:12.707 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:38:12.707 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:38:12.707 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:38:12.708 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:12.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:12.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:38:12.708 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:38:12.708 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:38:12.708 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:38:12.708 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:38:12.709 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:38:12.709 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:38:12.709 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:38:12.709 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:38:12.709 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:38:12.709 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:12.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:12.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:38:12.710 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:38:12.710 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:38:12.710 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:38:12.710 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:38:12.710 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:38:12.710 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:38:12.711 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:38:12.711 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:38:12.711 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:38:12.711 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:12.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:12.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:38:12.711 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:38:12.712 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:38:12.712 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:38:12.712 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:38:12.712 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:38:12.712 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:38:12.713 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:38:12.713 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:38:12.713 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:38:12.713 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:12.713 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:12.713 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:38:12.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:12.714 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:38:12.714 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:38:12.714 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:38:12.714 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:12.714 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:38:12.715 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:38:17.717 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:38:17.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:38:17.718 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:38:17.759 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:38:17.760 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:38:18.131 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:38:18.131 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:38:18.131 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:38:18.131 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:38:18.131 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:38:18.132 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:38:18.132 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:38:18.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:18.132 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:38:18.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:38:18.132 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:38:18.132 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:38:18.133 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:38:18.133 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:38:18.133 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:38:18.133 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:38:18.133 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:38:18.133 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:38:18.133 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:38:18.134 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:18.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:18.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:38:18.135 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:38:18.135 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:38:18.135 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:38:18.135 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:38:18.135 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:38:18.135 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:38:18.135 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:38:18.136 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:38:18.136 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:38:18.136 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:18.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:18.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:38:18.136 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:38:18.136 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:38:18.136 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:38:18.137 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:38:18.137 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:38:18.137 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:38:18.137 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:38:18.137 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:38:18.137 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:38:18.137 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:18.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:18.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:38:18.138 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:38:18.139 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:38:18.139 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:38:18.139 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:38:18.139 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:38:18.139 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:38:18.139 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:38:18.139 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:38:18.140 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:38:18.140 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:18.140 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:18.140 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:38:18.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:18.140 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:38:18.140 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:38:18.141 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:38:18.141 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:18.141 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:38:18.142 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:38:23.143 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:38:23.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:38:23.145 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:38:23.185 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:38:23.185 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:38:23.545 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:38:23.546 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:38:23.546 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:38:23.546 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:38:23.546 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:38:23.546 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:38:23.547 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:38:23.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:23.547 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:38:23.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:38:23.547 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:38:23.548 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:38:23.548 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:38:23.548 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:38:23.548 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:38:23.548 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:38:23.548 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:38:23.549 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:38:23.549 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:38:23.549 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:23.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:23.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:38:23.549 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:38:23.549 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:38:23.550 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:38:23.550 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:38:23.550 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:38:23.550 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:38:23.550 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:38:23.550 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:38:23.551 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:38:23.551 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:23.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:23.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:38:23.551 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:38:23.551 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:38:23.551 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:38:23.552 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:38:23.552 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:38:23.552 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:38:23.552 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:38:23.552 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:38:23.552 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:38:23.552 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:23.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:23.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:38:23.553 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:38:23.553 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:38:23.553 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:38:23.553 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:38:23.553 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:38:23.554 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:38:23.554 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:38:23.554 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:38:23.554 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:38:23.555 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:23.555 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:23.555 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:38:23.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:23.555 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:38:23.555 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:38:23.556 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:38:23.556 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:23.556 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:38:23.557 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:38:28.558 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:38:28.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:38:28.559 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:38:28.609 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:38:28.609 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:38:28.977 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:38:28.977 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:38:28.977 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:38:28.978 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:38:28.978 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:38:28.978 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:38:28.978 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:38:28.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:28.978 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:38:28.979 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:38:28.979 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:38:28.979 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:38:28.979 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:38:28.979 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:38:28.979 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:38:28.979 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:38:28.980 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:38:28.980 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:38:28.980 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:38:28.980 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:28.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:28.980 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:38:28.981 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:38:28.981 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:38:28.981 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:38:28.981 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:38:28.981 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:38:28.981 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:38:28.981 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:38:28.982 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:38:28.982 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:38:28.982 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:28.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:28.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:38:28.982 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:38:28.982 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:38:28.983 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:38:28.983 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:38:28.983 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:38:28.983 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:38:28.983 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:38:28.983 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:38:28.983 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:38:28.984 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:28.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:28.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:38:28.984 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:38:28.984 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:38:28.984 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:38:28.984 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:38:28.985 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:38:28.985 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:38:28.985 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:38:28.985 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:38:28.985 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:38:28.985 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:28.985 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:28.986 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:38:28.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:28.986 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:38:28.987 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:38:28.987 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:38:28.988 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:28.988 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:38:28.989 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:38:33.990 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:38:33.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:38:33.991 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:38:34.034 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:38:34.034 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:38:34.404 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:38:34.404 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:38:34.404 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:38:34.404 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:38:34.405 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:38:34.405 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:38:34.405 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:38:34.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:34.405 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:38:34.405 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:38:34.405 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:38:34.405 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:38:34.406 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:38:34.407 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:38:34.407 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:38:34.407 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:38:34.407 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:38:34.408 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:38:34.408 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:38:34.408 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:34.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:34.408 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:38:34.408 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:38:34.408 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:38:34.409 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:38:34.409 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:38:34.409 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:38:34.409 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:38:34.409 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:38:34.409 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:38:34.410 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:38:34.410 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:34.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:34.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:38:34.410 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:38:34.410 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:38:34.411 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:38:34.411 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:38:34.411 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:38:34.411 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:38:34.411 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:38:34.412 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:38:34.412 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:38:34.412 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:34.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:34.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:38:34.412 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:38:34.412 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:38:34.413 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:38:34.413 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:38:34.413 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:38:34.413 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:38:34.413 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:38:34.413 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:38:34.413 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:38:34.413 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:34.414 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:34.414 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:38:34.414 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:34.414 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:38:34.414 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:38:34.414 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:38:34.414 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:34.415 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:38:34.416 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:38:39.417 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:38:39.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:38:39.418 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:38:39.457 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:38:39.458 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:38:39.721 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:38:39.722 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:38:39.722 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:38:39.722 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:38:39.722 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:38:39.723 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:38:39.723 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:38:39.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:39.723 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:38:39.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:38:39.724 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:38:39.724 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:38:39.724 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:38:39.724 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:38:39.724 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:38:39.724 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:38:39.725 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:38:39.725 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:38:39.725 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:38:39.725 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:39.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:39.725 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:38:39.725 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:38:39.726 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:38:39.726 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:38:39.726 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:38:39.726 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:38:39.726 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:38:39.726 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:38:39.726 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:38:39.727 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:38:39.727 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:39.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:39.727 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:38:39.727 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:38:39.727 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:38:39.727 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:38:39.728 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:38:39.728 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:38:39.728 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:38:39.728 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:38:39.728 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:38:39.728 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:38:39.728 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:39.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:39.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:38:39.729 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:38:39.729 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:38:39.729 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:38:39.729 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:38:39.729 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:38:39.729 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:38:39.730 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:38:39.730 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:38:39.730 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:38:39.730 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:39.730 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:39.730 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:38:39.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:39.731 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:38:39.731 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:38:39.731 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:38:39.731 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:39.731 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:38:39.732 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:38:44.733 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:38:44.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:38:44.734 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:38:44.776 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:38:44.776 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:38:45.149 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:38:45.150 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:38:45.150 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:38:45.150 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:38:45.150 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:38:45.150 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:38:45.151 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:38:45.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:45.151 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:38:45.151 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:38:45.151 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:38:45.151 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:38:45.151 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:38:45.152 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:38:45.152 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:38:45.152 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:38:45.152 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:38:45.152 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:38:45.152 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:38:45.152 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:45.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:45.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:38:45.153 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:38:45.153 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:38:45.153 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:38:45.153 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:38:45.153 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:38:45.153 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:38:45.154 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:38:45.154 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:38:45.154 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:38:45.154 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:45.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:45.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:38:45.155 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:38:45.155 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:38:45.155 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:38:45.155 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:38:45.155 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:38:45.156 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:38:45.156 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:38:45.156 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:38:45.156 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:38:45.156 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:45.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:45.156 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:38:45.157 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:38:45.157 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:38:45.157 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:38:45.157 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:38:45.157 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:38:45.157 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:38:45.157 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:38:45.157 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:38:45.158 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:38:45.158 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:45.158 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:45.158 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:38:45.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:45.158 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:38:45.159 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:38:45.159 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:38:45.159 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:45.159 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:38:45.160 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:38:50.161 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:38:50.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:38:50.162 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:38:50.203 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:38:50.204 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:38:50.568 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:38:50.568 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:38:50.568 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:38:50.568 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:38:50.568 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:38:50.569 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:38:50.569 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:38:50.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:50.569 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:38:50.569 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:38:50.569 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:38:50.569 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:38:50.570 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:38:50.570 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:38:50.570 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:38:50.570 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:38:50.570 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:38:50.570 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:38:50.570 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:38:50.571 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:50.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:50.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:38:50.571 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:38:50.571 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:38:50.571 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:38:50.571 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:38:50.572 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:38:50.572 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:38:50.572 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:38:50.572 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:38:50.572 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:38:50.572 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:50.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:50.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:38:50.573 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:38:50.573 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:38:50.573 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:38:50.573 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:38:50.573 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:38:50.573 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:38:50.573 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:38:50.574 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:38:50.574 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:38:50.574 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:50.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:50.574 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:38:50.574 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:38:50.574 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:38:50.575 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:38:50.575 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:38:50.575 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:38:50.575 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:38:50.575 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:38:50.575 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:38:50.575 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:38:50.576 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:50.576 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:50.576 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:38:50.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:50.576 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:38:50.576 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:38:50.576 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:38:50.576 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:50.577 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:38:50.578 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:38:55.578 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:38:55.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:38:55.580 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:38:55.616 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:38:55.617 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:38:55.977 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:38:55.977 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:38:55.977 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:38:55.977 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:38:55.978 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:38:55.978 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:38:55.978 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:38:55.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:55.978 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:38:55.978 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:38:55.978 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:38:55.979 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:38:55.979 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:38:55.979 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:38:55.979 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:38:55.979 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:38:55.979 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:38:55.979 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:38:55.980 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:38:55.980 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:55.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:55.980 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:38:55.980 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:38:55.980 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:38:55.980 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:38:55.980 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:38:55.981 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:38:55.981 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:38:55.981 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:38:55.981 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:38:55.981 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:38:55.981 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:55.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:55.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:38:55.982 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:38:55.982 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:38:55.982 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:38:55.982 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:38:55.982 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:38:55.982 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:38:55.983 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:38:55.983 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:38:55.983 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:38:55.983 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:55.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:55.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:38:55.983 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:38:55.984 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:38:55.984 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:38:55.984 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:38:55.984 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:38:55.985 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:38:55.985 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:38:55.985 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:38:55.985 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:38:55.985 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:55.985 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:55.985 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:38:55.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:38:55.986 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:38:55.986 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:38:55.986 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:38:55.986 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:38:55.986 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:38:55.987 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:39:00.988 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:39:00.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:39:00.990 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:39:01.031 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:39:01.031 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:39:01.396 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:39:01.397 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:39:01.397 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:39:01.397 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:39:01.397 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:39:01.397 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:39:01.398 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:39:01.398 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:01.398 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:39:01.398 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:39:01.398 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:39:01.398 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:39:01.398 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:39:01.399 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:39:01.399 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:39:01.400 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:39:01.400 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:39:01.400 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:39:01.400 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:39:01.400 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:01.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:01.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:39:01.401 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:39:01.401 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:39:01.401 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:39:01.401 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:39:01.401 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:39:01.401 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:39:01.402 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:39:01.402 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:39:01.402 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:39:01.402 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:01.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:01.402 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:39:01.402 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:39:01.403 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:39:01.403 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:39:01.403 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:39:01.403 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:39:01.403 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:39:01.403 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:39:01.404 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:39:01.404 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:39:01.404 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:01.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:01.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:39:01.404 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:39:01.404 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:39:01.405 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:39:01.405 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:39:01.405 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:39:01.405 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:39:01.405 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:39:01.405 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:39:01.405 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:39:01.406 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:01.406 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:01.406 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:39:01.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:01.406 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:39:01.406 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:39:01.406 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:39:01.407 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:01.407 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:39:01.408 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:39:06.409 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:39:06.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:39:06.410 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:39:06.453 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:39:06.453 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:39:06.824 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:39:06.824 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:39:06.825 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:39:06.825 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:39:06.825 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:39:06.826 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:39:06.826 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:39:06.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:06.827 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:39:06.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:39:06.827 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:39:06.827 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:39:06.827 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:39:06.827 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:39:06.827 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:39:06.828 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:39:06.828 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:39:06.828 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:39:06.828 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:39:06.828 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:06.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:06.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:39:06.829 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:39:06.829 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:39:06.829 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:39:06.829 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:39:06.829 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:39:06.829 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:39:06.829 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:39:06.829 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:39:06.830 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:39:06.830 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:06.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:06.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:39:06.830 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:39:06.830 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:39:06.830 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:39:06.831 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:39:06.831 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:39:06.831 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:39:06.831 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:39:06.831 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:39:06.831 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:39:06.831 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:06.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:06.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:39:06.832 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:39:06.832 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:39:06.832 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:39:06.832 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:39:06.832 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:39:06.833 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:39:06.833 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:39:06.833 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:39:06.833 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:39:06.833 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:06.833 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:06.833 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:39:06.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:06.834 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:39:06.834 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:39:06.834 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:39:06.834 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:06.834 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:39:06.835 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:39:11.837 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:39:11.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:39:11.838 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:39:11.879 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:39:11.879 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:39:12.252 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:39:12.252 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:39:12.252 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:39:12.252 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:39:12.253 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:39:12.253 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:39:12.253 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:39:12.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:12.253 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:39:12.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:39:12.253 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:39:12.254 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:39:12.254 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:39:12.254 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:39:12.254 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:39:12.254 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:39:12.254 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:39:12.254 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:39:12.255 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:39:12.255 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:12.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:12.255 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:39:12.255 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:39:12.255 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:39:12.255 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:39:12.256 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:39:12.256 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:39:12.256 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:39:12.256 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:39:12.256 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:39:12.256 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:39:12.256 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:12.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:12.257 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:39:12.257 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:39:12.257 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:39:12.257 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:39:12.257 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:39:12.257 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:39:12.258 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:39:12.258 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:39:12.258 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:39:12.258 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:39:12.258 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:12.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:12.258 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:39:12.259 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:39:12.259 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:39:12.259 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:39:12.259 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:39:12.259 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:39:12.259 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:39:12.259 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:39:12.260 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:39:12.260 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:39:12.260 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:12.260 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:12.260 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:39:12.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:12.260 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:39:12.261 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:39:12.261 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:39:12.261 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:12.261 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:39:12.262 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:39:17.263 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:39:17.264 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:39:17.266 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:39:17.307 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:39:17.308 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:39:17.676 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:39:17.676 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:39:17.676 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:39:17.676 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:39:17.676 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:39:17.677 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:39:17.677 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:39:17.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:17.677 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:39:17.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:39:17.677 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:39:17.677 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:39:17.678 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:39:17.678 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:39:17.678 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:39:17.678 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:39:17.678 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:39:17.679 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:39:17.679 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:39:17.680 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:17.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:17.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:39:17.680 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:39:17.681 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:39:17.681 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:39:17.681 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:39:17.681 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:39:17.681 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:39:17.681 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:39:17.681 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:39:17.682 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:39:17.682 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:17.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:17.682 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:39:17.682 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:39:17.682 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:39:17.682 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:39:17.683 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:39:17.683 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:39:17.683 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:39:17.683 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:39:17.683 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:39:17.683 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:39:17.683 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:17.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:17.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:39:17.684 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:39:17.684 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:39:17.684 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:39:17.684 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:39:17.684 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:39:17.685 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:39:17.685 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:39:17.685 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:39:17.685 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:39:17.685 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:17.685 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:17.685 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:39:17.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:17.686 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:39:17.686 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:39:17.686 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:39:17.686 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:17.686 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:39:17.687 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:39:22.688 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:39:22.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:39:22.689 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:39:22.731 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:39:22.731 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:39:22.993 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:39:22.994 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:39:22.994 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:39:22.994 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:39:22.995 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:39:22.995 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:39:22.995 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:39:22.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:22.995 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:39:22.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:39:22.996 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:39:22.996 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:39:22.996 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:39:22.996 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:39:22.996 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:39:22.996 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:39:22.996 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:39:22.997 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:39:22.997 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:39:22.997 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:22.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:22.997 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:39:22.997 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:39:22.997 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:39:22.998 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:39:22.998 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:39:22.998 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:39:22.998 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:39:22.998 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:39:22.998 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:39:22.999 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:39:22.999 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:22.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:22.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:39:22.999 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:39:22.999 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:39:22.999 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:39:22.999 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:39:23.000 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:39:23.000 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:39:23.000 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:39:23.000 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:39:23.000 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:39:23.000 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:23.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:23.001 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:39:23.001 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:39:23.001 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:39:23.001 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:39:23.001 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:39:23.001 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:39:23.001 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:39:23.002 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:39:23.002 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:39:23.002 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:39:23.002 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:23.002 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:23.002 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:39:23.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:23.003 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:39:23.003 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:39:23.003 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:39:23.003 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:23.003 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:39:23.004 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:39:28.005 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:39:28.006 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:39:28.007 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:39:28.049 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:39:28.049 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:39:28.416 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:39:28.416 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:39:28.416 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:39:28.417 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:39:28.417 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:39:28.417 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:39:28.417 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:39:28.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:28.417 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:39:28.417 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:39:28.418 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:39:28.418 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:39:28.418 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:39:28.418 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:39:28.418 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:39:28.418 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:39:28.418 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:39:28.419 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:39:28.419 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:39:28.419 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:28.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:28.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:39:28.419 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:39:28.419 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:39:28.420 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:39:28.420 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:39:28.420 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:39:28.420 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:39:28.420 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:39:28.420 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:39:28.420 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:39:28.421 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:28.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:28.421 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:39:28.421 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:39:28.421 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:39:28.421 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:39:28.421 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:39:28.422 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:39:28.422 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:39:28.422 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:39:28.422 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:39:28.422 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:39:28.422 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:28.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:28.423 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:39:28.423 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:39:28.423 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:39:28.423 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:39:28.423 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:39:28.423 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:39:28.423 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:39:28.423 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:39:28.424 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:39:28.424 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:39:28.424 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:28.424 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:28.424 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:39:28.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:28.424 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:39:28.425 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:39:28.425 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:39:28.425 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:28.425 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:39:28.426 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:39:33.427 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:39:33.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:39:33.428 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:39:33.468 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:39:33.468 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:39:33.832 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:39:33.833 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:39:33.833 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:39:33.833 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:39:33.833 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:39:33.833 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:39:33.833 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:39:33.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:33.834 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:39:33.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:39:33.834 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:39:33.834 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:39:33.834 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:39:33.835 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:39:33.835 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:39:33.835 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:39:33.835 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:39:33.835 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:39:33.835 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:39:33.835 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:33.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:33.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:39:33.836 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:39:33.837 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:39:33.837 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:39:33.837 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:39:33.837 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:39:33.837 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:39:33.837 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:39:33.838 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:39:33.838 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:39:33.839 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:33.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:33.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:39:33.839 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:39:33.839 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:39:33.839 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:39:33.840 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:39:33.840 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:39:33.840 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:39:33.841 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:39:33.841 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:39:33.841 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:39:33.841 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:33.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:33.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:39:33.841 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:39:33.842 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:39:33.842 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:39:33.842 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:39:33.842 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:39:33.842 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:39:33.842 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:39:33.842 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:39:33.843 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:39:33.843 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:33.843 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:33.843 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:39:33.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:33.843 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:39:33.843 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:39:33.844 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:39:33.844 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:33.844 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:39:33.845 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:39:38.846 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:39:38.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:39:38.848 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:39:38.889 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:39:38.889 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:39:39.259 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:39:39.259 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:39:39.260 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:39:39.260 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:39:39.260 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:39:39.260 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:39:39.260 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:39:39.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:39.261 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:39:39.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:39:39.261 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:39:39.261 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:39:39.261 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:39:39.261 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:39:39.261 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:39:39.262 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:39:39.262 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:39:39.262 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:39:39.262 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:39:39.262 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:39.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:39.262 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:39:39.263 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:39:39.263 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:39:39.263 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:39:39.263 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:39:39.263 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:39:39.263 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:39:39.263 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:39:39.264 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:39:39.264 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:39:39.264 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:39.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:39.264 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:39:39.264 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:39:39.264 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:39:39.264 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:39:39.265 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:39:39.265 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:39:39.265 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:39:39.265 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:39:39.265 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:39:39.265 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:39:39.265 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:39.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:39.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:39:39.266 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:39:39.266 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:39:39.266 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:39:39.266 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:39:39.266 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:39:39.267 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:39:39.267 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:39:39.267 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:39:39.267 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:39:39.267 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:39.267 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:39.267 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:39:39.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:39.268 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:39:39.268 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:39:39.268 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:39:39.268 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:39.268 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:39:39.269 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:39:44.270 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:39:44.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:39:44.272 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:39:44.316 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:39:44.316 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:39:44.577 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:39:44.578 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:39:44.578 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:39:44.578 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:39:44.578 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:39:44.578 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:39:44.578 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:39:44.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:44.579 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:39:44.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:39:44.579 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:39:44.579 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:39:44.579 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:39:44.579 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:39:44.580 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:39:44.580 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:39:44.580 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:39:44.580 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:39:44.580 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:39:44.580 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:44.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:44.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:39:44.581 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:39:44.581 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:39:44.581 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:39:44.581 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:39:44.581 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:39:44.581 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:39:44.581 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:39:44.582 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:39:44.582 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:39:44.582 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:44.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:44.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:39:44.582 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:39:44.582 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:39:44.583 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:39:44.583 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:39:44.583 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:39:44.583 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:39:44.583 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:39:44.583 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:39:44.583 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:39:44.584 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:44.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:44.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:39:44.584 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:39:44.584 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:39:44.584 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:39:44.584 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:39:44.585 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:39:44.585 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:39:44.585 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:39:44.585 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:39:44.585 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:39:44.585 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:44.585 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:44.586 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:39:44.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:44.586 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:39:44.586 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:39:44.586 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:39:44.586 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:44.586 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:39:44.587 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:39:49.589 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:39:49.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:39:49.590 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:39:49.631 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:39:49.631 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:39:49.885 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:39:49.885 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:39:49.885 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:39:49.885 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:39:49.885 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:39:49.886 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:39:49.886 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:39:49.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:49.886 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:39:49.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:39:49.886 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:39:49.887 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:39:49.887 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:39:49.887 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:39:49.887 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:39:49.887 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:39:49.887 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:39:49.887 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:39:49.888 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:39:49.888 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:49.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:49.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:39:49.889 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:39:49.889 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:39:49.889 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:39:49.889 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:39:49.889 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:39:49.889 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:39:49.890 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:39:49.890 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:39:49.890 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:39:49.890 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:49.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:49.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:39:49.890 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:39:49.891 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:39:49.891 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:39:49.891 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:39:49.891 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:39:49.891 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:39:49.891 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:39:49.891 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:39:49.892 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:39:49.892 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:49.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:49.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:39:49.892 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:39:49.893 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:39:49.893 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:39:49.893 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:39:49.893 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:39:49.893 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:39:49.893 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:39:49.893 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:39:49.894 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:39:49.894 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:49.895 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:49.895 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:39:49.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:49.895 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:39:49.895 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:39:49.895 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:39:49.895 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:49.896 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:39:49.897 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:39:54.898 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:39:54.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:39:54.899 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:39:54.940 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:39:54.941 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:39:55.303 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:39:55.303 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:39:55.304 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:39:55.304 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:39:55.304 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:39:55.304 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:39:55.305 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:39:55.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:55.305 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:39:55.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:39:55.305 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:39:55.305 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:39:55.306 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:39:55.306 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:39:55.306 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:39:55.306 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:39:55.306 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:39:55.306 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:39:55.307 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:39:55.307 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:55.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:55.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:39:55.307 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:39:55.307 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:39:55.308 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:39:55.308 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:39:55.308 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:39:55.308 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:39:55.309 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:39:55.309 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:39:55.309 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:39:55.309 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:55.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:55.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:39:55.309 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:39:55.310 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:39:55.310 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:39:55.310 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:39:55.310 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:39:55.310 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:39:55.311 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:39:55.311 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:39:55.311 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:39:55.311 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:55.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:55.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:39:55.312 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:39:55.312 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:39:55.312 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:39:55.312 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:39:55.312 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:39:55.312 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:39:55.312 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:39:55.313 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:39:55.313 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:39:55.313 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:55.313 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:55.314 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:39:55.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:39:55.314 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:39:55.314 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:39:55.315 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:39:55.315 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:39:55.315 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:39:55.317 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:40:00.318 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:40:00.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:40:00.320 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:40:00.360 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:40:00.360 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:40:00.727 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:40:00.727 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:40:00.728 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:40:00.728 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:40:00.728 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:40:00.728 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:40:00.728 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:40:00.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:00.729 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:40:00.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:40:00.729 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:40:00.729 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:40:00.729 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:40:00.729 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:40:00.729 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:40:00.730 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:40:00.730 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:40:00.730 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:40:00.730 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:40:00.730 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:00.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:00.730 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:40:00.731 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:40:00.731 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:40:00.731 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:40:00.731 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:40:00.731 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:40:00.731 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:40:00.731 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:40:00.732 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:40:00.732 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:40:00.732 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:00.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:00.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:40:00.732 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:40:00.733 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:40:00.733 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:40:00.733 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:40:00.733 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:40:00.733 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:40:00.733 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:40:00.733 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:40:00.734 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:40:00.734 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:00.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:00.734 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:40:00.734 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:40:00.734 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:40:00.734 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:40:00.735 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:40:00.735 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:40:00.735 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:40:00.735 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:40:00.735 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:40:00.735 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:40:00.735 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:00.736 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:00.736 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:40:00.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:00.736 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:40:00.736 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:40:00.736 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:40:00.736 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:00.737 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:40:00.738 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:40:05.740 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:40:05.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:40:05.741 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:40:05.782 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:40:05.782 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:40:06.162 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:40:06.164 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:40:06.165 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:40:06.165 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:40:06.165 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:40:06.165 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:40:06.165 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:40:06.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:06.166 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:40:06.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:40:06.166 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:40:06.166 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:40:06.166 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:40:06.166 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:40:06.166 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:40:06.167 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:40:06.167 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:40:06.167 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:40:06.167 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:40:06.167 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:06.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:06.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:40:06.168 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:40:06.168 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:40:06.168 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:40:06.168 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:40:06.168 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:40:06.168 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:40:06.169 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:40:06.169 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:40:06.169 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:40:06.169 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:06.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:06.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:40:06.169 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:40:06.170 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:40:06.170 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:40:06.170 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:40:06.170 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:40:06.170 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:40:06.170 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:40:06.171 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:40:06.171 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:40:06.172 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:06.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:06.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:40:06.172 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:40:06.172 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:40:06.173 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:40:06.173 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:40:06.173 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:40:06.173 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:40:06.173 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:40:06.173 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:40:06.173 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:40:06.174 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:06.174 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:06.174 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:40:06.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:06.174 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:40:06.174 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:40:06.174 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:40:06.175 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:06.175 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:40:06.175 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:40:11.176 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:40:11.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:40:11.178 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:40:11.221 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:40:11.221 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:40:11.594 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:40:11.594 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:40:11.595 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:40:11.595 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:40:11.595 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:40:11.595 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:40:11.595 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:40:11.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:11.596 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:40:11.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:40:11.596 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:40:11.596 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:40:11.596 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:40:11.596 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:40:11.596 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:40:11.597 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:40:11.597 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:40:11.597 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:40:11.597 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:40:11.597 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:11.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:11.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:40:11.598 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:40:11.598 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:40:11.598 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:40:11.598 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:40:11.598 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:40:11.598 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:40:11.599 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:40:11.599 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:40:11.599 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:40:11.599 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:11.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:11.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:40:11.599 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:40:11.600 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:40:11.600 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:40:11.600 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:40:11.600 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:40:11.600 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:40:11.600 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:40:11.600 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:40:11.601 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:40:11.601 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:11.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:11.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:40:11.601 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:40:11.601 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:40:11.601 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:40:11.602 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:40:11.602 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:40:11.602 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:40:11.602 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:40:11.602 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:40:11.602 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:40:11.603 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:11.603 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:11.603 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:40:11.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:11.603 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:40:11.603 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:40:11.603 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:40:11.604 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:11.604 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:40:11.605 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:40:16.606 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:40:16.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:40:16.607 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:40:16.649 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:40:16.649 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:40:17.012 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:40:17.013 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:40:17.013 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:40:17.013 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:40:17.014 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:40:17.014 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:40:17.014 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:40:17.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:17.015 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:40:17.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:40:17.015 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:40:17.015 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:40:17.015 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:40:17.016 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:40:17.016 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:40:17.016 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:40:17.016 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:40:17.017 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:40:17.017 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:40:17.017 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:17.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:17.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:40:17.018 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:40:17.018 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:40:17.018 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:40:17.020 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:40:17.020 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:40:17.021 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:40:17.021 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:40:17.021 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:40:17.022 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:40:17.022 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:17.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:17.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:40:17.023 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:40:17.023 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:40:17.023 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:40:17.023 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:40:17.024 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:40:17.024 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:40:17.024 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:40:17.024 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:40:17.024 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:40:17.024 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:17.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:17.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:40:17.025 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:40:17.025 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:40:17.025 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:40:17.026 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:40:17.026 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:40:17.026 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:40:17.026 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:40:17.026 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:40:17.027 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:40:17.027 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:17.027 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:17.028 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:40:17.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:17.028 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:40:17.029 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:40:17.029 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:40:17.029 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:17.029 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:40:17.031 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:40:22.032 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:40:22.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:40:22.033 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:40:22.075 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:40:22.075 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:40:22.444 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:40:22.445 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:40:22.445 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:40:22.445 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:40:22.445 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:40:22.445 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:40:22.446 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:40:22.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:22.446 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:40:22.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:40:22.446 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:40:22.446 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:40:22.447 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:40:22.447 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:40:22.447 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:40:22.447 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:40:22.447 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:40:22.447 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:40:22.448 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:40:22.448 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:22.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:22.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:40:22.448 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:40:22.449 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:40:22.449 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:40:22.449 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:40:22.449 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:40:22.449 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:40:22.449 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:40:22.449 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:40:22.450 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:40:22.450 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:22.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:22.450 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:40:22.450 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:40:22.450 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:40:22.450 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:40:22.451 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:40:22.451 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:40:22.451 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:40:22.451 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:40:22.451 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:40:22.451 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:40:22.451 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:22.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:22.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:40:22.452 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:40:22.452 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:40:22.452 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:40:22.452 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:40:22.452 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:40:22.453 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:40:22.453 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:40:22.453 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:40:22.453 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:40:22.453 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:22.453 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:22.453 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:40:22.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:22.454 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:40:22.454 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:40:22.454 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:40:22.454 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:22.454 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:40:22.455 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:40:27.457 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:40:27.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:40:27.458 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:40:27.501 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:40:27.501 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:40:27.865 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:40:27.865 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:40:27.865 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:40:27.865 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:40:27.866 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:40:27.866 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:40:27.866 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:40:27.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:27.866 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:40:27.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:40:27.866 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:40:27.867 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:40:27.867 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:40:27.867 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:40:27.867 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:40:27.867 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:40:27.867 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:40:27.867 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:40:27.868 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:40:27.868 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:27.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:27.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:40:27.870 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:40:27.870 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:40:27.870 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:40:27.870 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:40:27.871 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:40:27.871 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:40:27.871 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:40:27.871 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:40:27.871 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:40:27.871 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:27.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:27.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:40:27.872 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:40:27.872 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:40:27.872 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:40:27.872 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:40:27.872 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:40:27.873 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:40:27.873 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:40:27.873 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:40:27.873 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:40:27.873 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:27.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:27.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:40:27.873 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:40:27.874 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:40:27.874 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:40:27.874 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:40:27.874 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:40:27.874 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:40:27.874 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:40:27.875 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:40:27.875 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:40:27.875 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:27.875 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:27.875 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:40:27.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:27.875 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:40:27.876 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:40:27.876 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:40:27.876 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:27.876 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:40:27.877 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:40:32.878 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:40:32.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:40:32.881 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:40:32.924 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:40:32.925 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:40:33.298 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:40:33.298 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:40:33.298 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:40:33.299 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:40:33.299 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:40:33.299 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:40:33.299 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:40:33.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:33.299 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:40:33.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:40:33.301 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:40:33.301 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:40:33.301 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:40:33.301 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:40:33.301 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:40:33.302 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:40:33.302 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:40:33.302 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:40:33.302 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:40:33.302 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:33.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:33.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:40:33.303 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:40:33.303 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:40:33.303 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:40:33.303 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:40:33.303 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:40:33.303 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:40:33.303 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:40:33.303 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:40:33.304 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:40:33.304 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:33.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:33.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:40:33.304 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:40:33.304 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:40:33.304 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:40:33.305 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:40:33.305 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:40:33.305 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:40:33.305 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:40:33.305 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:40:33.305 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:40:33.305 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:33.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:33.306 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:40:33.306 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:40:33.306 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:40:33.306 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:40:33.306 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:40:33.306 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:40:33.306 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:40:33.307 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:40:33.307 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:40:33.307 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:40:33.307 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:33.307 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:33.307 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:40:33.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:33.308 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:40:33.308 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:40:33.308 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:40:33.308 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:33.308 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:40:33.310 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:40:38.312 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:40:38.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:40:38.313 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:40:38.354 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:40:38.354 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:40:38.719 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:40:38.720 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:40:38.720 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:40:38.720 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:40:38.720 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:40:38.720 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:40:38.720 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:40:38.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:38.721 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:40:38.721 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:40:38.721 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:40:38.721 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:40:38.721 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:40:38.721 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:40:38.722 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:40:38.722 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:40:38.722 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:40:38.722 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:40:38.722 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:40:38.722 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:38.722 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:38.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:40:38.723 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:40:38.723 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:40:38.723 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:40:38.723 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:40:38.723 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:40:38.723 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:40:38.724 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:40:38.724 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:40:38.724 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:40:38.724 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:38.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:38.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:40:38.724 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:40:38.724 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:40:38.725 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:40:38.725 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:40:38.725 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:40:38.725 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:40:38.725 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:40:38.725 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:40:38.725 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:40:38.726 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:38.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:38.726 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:40:38.726 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:40:38.726 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:40:38.726 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:40:38.726 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:40:38.727 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:40:38.727 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:40:38.727 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:40:38.727 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:40:38.727 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:40:38.727 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:38.727 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:38.727 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:40:38.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:38.728 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:40:38.728 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:40:38.728 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:40:38.728 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:38.728 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:40:38.729 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:40:43.730 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:40:43.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:40:43.731 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:40:43.773 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:40:43.773 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:40:44.143 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:40:44.143 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:40:44.143 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:40:44.144 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:40:44.144 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:40:44.144 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:40:44.144 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:40:44.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:44.144 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:40:44.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:40:44.145 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:40:44.145 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:40:44.145 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:40:44.145 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:40:44.145 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:40:44.145 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:40:44.145 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:40:44.146 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:40:44.146 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:40:44.146 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:44.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:44.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:40:44.146 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:40:44.147 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:40:44.147 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:40:44.147 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:40:44.147 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:40:44.147 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:40:44.147 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:40:44.147 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:40:44.147 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:40:44.148 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:44.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:44.148 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:40:44.148 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:40:44.148 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:40:44.148 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:40:44.148 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:40:44.149 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:40:44.149 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:40:44.149 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:40:44.149 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:40:44.149 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:40:44.149 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:44.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:44.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:40:44.150 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:40:44.150 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:40:44.150 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:40:44.150 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:40:44.150 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:40:44.150 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:40:44.151 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:40:44.151 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:40:44.151 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:40:44.151 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:44.151 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:44.151 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:40:44.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:44.151 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:40:44.152 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:40:44.152 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:40:44.152 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:44.152 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:40:44.153 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:40:49.155 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:40:49.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:40:49.156 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:40:49.199 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:40:49.199 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:40:49.567 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:40:49.568 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:40:49.568 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:40:49.568 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:40:49.568 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:40:49.568 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:40:49.568 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:40:49.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:49.569 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:40:49.569 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:40:49.569 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:40:49.569 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:40:49.569 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:40:49.569 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:40:49.570 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:40:49.570 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:40:49.570 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:40:49.570 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:40:49.570 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:40:49.570 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:49.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:49.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:40:49.571 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:40:49.571 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:40:49.571 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:40:49.571 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:40:49.571 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:40:49.571 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:40:49.572 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:40:49.572 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:40:49.572 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:40:49.572 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:49.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:49.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:40:49.572 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:40:49.573 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:40:49.573 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:40:49.573 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:40:49.573 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:40:49.573 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:40:49.573 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:40:49.573 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:40:49.574 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:40:49.574 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:49.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:49.574 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:40:49.574 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:40:49.574 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:40:49.574 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:40:49.575 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:40:49.575 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:40:49.575 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:40:49.575 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:40:49.575 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:40:49.575 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:40:49.575 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:49.576 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:49.576 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:40:49.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:49.576 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:40:49.576 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:40:49.576 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:40:49.576 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:49.576 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:40:49.577 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:40:54.579 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:40:54.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:40:54.580 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:40:54.623 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:40:54.623 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:40:54.879 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:40:54.879 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:40:54.879 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:40:54.880 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:40:54.880 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:40:54.880 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:40:54.880 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:40:54.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:54.880 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:40:54.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:40:54.881 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:40:54.881 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:40:54.881 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:40:54.881 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:40:54.881 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:40:54.881 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:40:54.881 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:40:54.882 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:40:54.882 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:40:54.882 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:54.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:54.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:40:54.882 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:40:54.882 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:40:54.883 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:40:54.883 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:40:54.883 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:40:54.883 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:40:54.883 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:40:54.883 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:40:54.883 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:40:54.884 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:54.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:54.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:40:54.884 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:40:54.884 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:40:54.884 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:40:54.884 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:40:54.885 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:40:54.885 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:40:54.885 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:40:54.885 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:40:54.885 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:40:54.885 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:54.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:54.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:40:54.886 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:40:54.886 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:40:54.886 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:40:54.886 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:40:54.886 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:40:54.886 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:40:54.886 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:40:54.887 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:40:54.887 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:40:54.887 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:54.887 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:54.887 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:40:54.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:40:54.887 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:40:54.888 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:40:54.888 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:40:54.888 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:40:54.888 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:40:54.889 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:40:59.890 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:40:59.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:40:59.891 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:40:59.934 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:40:59.934 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:41:00.406 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:41:00.407 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:41:00.407 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:41:00.407 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:41:00.408 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:41:00.408 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:41:00.408 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:41:00.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:00.409 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:41:00.409 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:00.409 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:41:00.410 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:00.410 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:00.410 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:41:00.410 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:00.410 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:41:00.410 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:00.411 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:41:00.411 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:41:00.411 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:00.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:00.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:00.411 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:41:00.411 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:00.411 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:00.412 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:41:00.412 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:00.412 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:41:00.412 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:00.412 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:41:00.412 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:41:00.412 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:00.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:00.413 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:00.413 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:41:00.413 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:00.413 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:00.413 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:41:00.413 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:00.414 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:41:00.414 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:00.414 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:41:00.414 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:41:00.414 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:00.414 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:00.414 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:00.415 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:41:00.415 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:00.415 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:00.415 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:41:00.415 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:00.415 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:41:00.415 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:00.416 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:41:00.416 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:41:00.416 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:00.416 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:00.417 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:41:00.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:00.417 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:41:00.417 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:41:00.417 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:41:00.418 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:00.418 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:41:00.420 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:41:05.421 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:41:05.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:41:05.422 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:41:05.471 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:41:05.471 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:41:05.733 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:41:05.733 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:41:05.733 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:41:05.734 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:41:05.734 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:41:05.734 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:41:05.734 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:41:05.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:05.734 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:41:05.734 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:05.735 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:41:05.735 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:05.735 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:05.735 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:41:05.735 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:05.735 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:41:05.735 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:05.736 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:41:05.736 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:41:05.736 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:05.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:05.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:05.736 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:41:05.736 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:05.736 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:05.737 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:41:05.737 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:05.737 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:41:05.737 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:05.737 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:41:05.737 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:41:05.737 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:05.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:05.738 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:05.738 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:41:05.738 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:05.738 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:05.738 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:41:05.738 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:05.739 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:41:05.739 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:05.739 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:41:05.739 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:41:05.739 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:05.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:05.740 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:05.740 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:41:05.740 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:05.740 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:05.740 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:41:05.740 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:05.740 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:41:05.741 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:05.741 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:41:05.741 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:41:05.741 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:05.741 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:05.741 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:41:05.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:05.742 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:41:05.742 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:41:05.742 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:41:05.742 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:05.742 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:41:05.743 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:41:10.745 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:41:10.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:41:10.746 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:41:10.785 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:41:10.786 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:41:11.053 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:41:11.053 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:41:11.054 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:41:11.054 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:41:11.054 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:41:11.054 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:41:11.054 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:41:11.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:11.057 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:41:11.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:11.057 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:41:11.057 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:11.057 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:11.057 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:41:11.057 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:11.058 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:41:11.058 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:11.058 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:41:11.058 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:41:11.058 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:11.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:11.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:11.059 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:41:11.059 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:11.059 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:11.059 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:41:11.059 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:11.059 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:41:11.059 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:11.059 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:41:11.060 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:41:11.061 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:11.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:11.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:11.061 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:41:11.061 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:11.061 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:11.061 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:41:11.062 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:11.062 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:41:11.062 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:11.062 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:41:11.062 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:41:11.062 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:11.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:11.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:11.063 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:41:11.063 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:11.063 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:11.063 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:41:11.063 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:11.063 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:41:11.064 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:11.064 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:41:11.064 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:41:11.064 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:11.064 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:11.064 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:41:11.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:11.064 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:41:11.065 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:41:11.065 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:41:11.065 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:11.065 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:41:11.066 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:41:16.067 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:41:16.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:41:16.069 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:41:16.108 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:41:16.108 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:41:16.479 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:41:16.479 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:41:16.479 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:41:16.479 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:41:16.480 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:41:16.480 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:41:16.480 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:41:16.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:16.480 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:41:16.481 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:16.481 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:41:16.481 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:16.481 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:16.481 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:41:16.481 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:16.481 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:41:16.482 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:16.482 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:41:16.482 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:41:16.482 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:16.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:16.482 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:16.482 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:41:16.483 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:16.483 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:16.483 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:41:16.483 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:16.483 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:41:16.483 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:16.483 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:41:16.484 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:41:16.484 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:16.484 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:16.484 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:16.484 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:41:16.484 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:16.484 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:16.484 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:41:16.485 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:16.485 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:41:16.485 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:16.485 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:41:16.485 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:41:16.486 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:16.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:16.486 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:16.486 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:41:16.486 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:16.487 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:16.488 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:41:16.488 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:16.488 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:41:16.488 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:16.488 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:41:16.489 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:41:16.489 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:16.489 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:16.489 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:41:16.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:16.489 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:41:16.489 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:41:16.489 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:41:16.490 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:16.490 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:41:16.491 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:41:21.492 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:41:21.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:41:21.494 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:41:21.535 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:41:21.536 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:41:21.901 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:41:21.901 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:41:21.902 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:41:21.902 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:41:21.902 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:41:21.902 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:41:21.902 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:41:21.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:21.903 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:41:21.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:21.903 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:41:21.903 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:21.903 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:21.903 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:41:21.903 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:21.903 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:41:21.904 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:21.904 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:41:21.904 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:41:21.904 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:21.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:21.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:21.904 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:41:21.905 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:21.905 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:21.905 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:41:21.905 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:21.905 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:41:21.905 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:21.905 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:41:21.906 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:41:21.906 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:21.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:21.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:21.906 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:41:21.906 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:21.907 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:21.907 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:41:21.907 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:21.907 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:41:21.907 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:21.907 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:41:21.907 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:41:21.908 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:21.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:21.908 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:21.908 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:41:21.908 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:21.908 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:21.908 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:41:21.908 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:21.909 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:41:21.909 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:21.909 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:41:21.909 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:41:21.909 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:21.909 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:21.909 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:41:21.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:21.910 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:41:21.910 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:41:21.910 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:41:21.910 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:21.910 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:41:21.911 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:41:26.912 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:41:26.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:41:26.914 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:41:26.958 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:41:26.958 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:41:27.325 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:41:27.325 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:41:27.325 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:41:27.326 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:41:27.326 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:41:27.326 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:41:27.326 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:41:27.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:27.326 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:41:27.327 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:27.327 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:41:27.327 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:27.327 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:27.327 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:41:27.327 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:27.327 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:41:27.328 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:27.328 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:41:27.328 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:41:27.329 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:27.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:27.329 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:27.329 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:41:27.329 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:27.329 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:27.329 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:41:27.330 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:27.330 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:41:27.330 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:27.330 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:41:27.330 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:41:27.330 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:27.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:27.330 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:27.331 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:41:27.331 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:27.331 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:27.331 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:41:27.331 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:27.331 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:41:27.331 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:27.332 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:41:27.332 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:41:27.332 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:27.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:27.332 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:27.332 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:41:27.332 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:27.333 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:27.333 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:41:27.333 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:27.333 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:41:27.333 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:27.333 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:41:27.333 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:41:27.333 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:27.334 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:27.334 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:41:27.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:27.335 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:41:27.335 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:41:27.335 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:41:27.335 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:27.335 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:41:27.336 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:41:32.339 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:41:32.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:41:32.340 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:41:32.381 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:41:32.382 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:41:32.654 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:41:32.654 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:41:32.654 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:41:32.654 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:41:32.655 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:41:32.655 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:41:32.655 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:41:32.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:32.655 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:41:32.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:32.655 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:41:32.656 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:32.656 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:32.656 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:41:32.656 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:32.656 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:41:32.656 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:32.656 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:41:32.657 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:41:32.657 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:32.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:32.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:32.657 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:41:32.657 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:32.657 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:32.657 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:41:32.658 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:32.658 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:41:32.658 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:32.658 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:41:32.658 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:41:32.658 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:32.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:32.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:32.659 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:41:32.659 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:32.659 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:32.659 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:41:32.659 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:32.659 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:41:32.660 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:32.660 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:41:32.660 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:41:32.660 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:32.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:32.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:32.660 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:41:32.660 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:32.661 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:32.661 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:41:32.661 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:32.661 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:41:32.661 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:32.661 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:41:32.661 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:41:32.662 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:32.662 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:32.662 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:41:32.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:32.662 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:41:32.662 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:41:32.662 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:41:32.663 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:32.663 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:41:32.664 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:41:37.665 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:41:37.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:41:37.667 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:41:37.707 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:41:37.708 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:41:38.072 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:41:38.073 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:41:38.073 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:41:38.073 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:41:38.074 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:41:38.074 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:41:38.074 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:41:38.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:38.075 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:41:38.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:38.076 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:41:38.076 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:38.076 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:38.076 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:41:38.077 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:38.077 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:41:38.077 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:38.078 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:41:38.078 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:41:38.078 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:38.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:38.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:38.081 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:41:38.081 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:38.081 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:38.082 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:41:38.082 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:38.082 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:41:38.083 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:38.083 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:41:38.083 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:41:38.084 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:38.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:38.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:38.085 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:41:38.085 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:38.085 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:38.086 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:41:38.086 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:38.086 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:41:38.086 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:38.087 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:41:38.087 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:41:38.087 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:38.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:38.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:38.088 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:41:38.089 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:38.089 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:38.090 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:41:38.092 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:38.092 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:41:38.092 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:38.093 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:41:38.093 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:41:38.093 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:38.094 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:38.094 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:41:38.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:38.095 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:41:38.095 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:41:38.095 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:41:38.095 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:38.096 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:41:38.098 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:41:43.100 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:41:43.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:41:43.102 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:41:43.146 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:41:43.146 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:41:43.508 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:41:43.509 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:41:43.509 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:41:43.509 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:41:43.510 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:41:43.510 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:41:43.510 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:41:43.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:43.511 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:41:43.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:43.512 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:41:43.512 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:43.512 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:43.513 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:41:43.513 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:43.513 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:41:43.514 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:43.514 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:41:43.514 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:41:43.515 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:43.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:43.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:43.516 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:41:43.517 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:43.517 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:43.517 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:41:43.517 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:43.517 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:41:43.517 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:43.517 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:41:43.518 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:41:43.518 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:43.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:43.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:43.519 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:41:43.519 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:43.519 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:43.520 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:41:43.521 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:43.521 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:41:43.521 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:43.521 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:41:43.521 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:41:43.521 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:43.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:43.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:43.522 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:41:43.522 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:43.522 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:43.522 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:41:43.522 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:43.522 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:41:43.523 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:43.523 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:41:43.523 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:41:43.523 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:43.523 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:43.523 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:41:43.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:43.524 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:41:43.524 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:41:43.524 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:41:43.524 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:43.524 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:41:43.525 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:41:48.528 INFO:tasks.rgw_multisite_tests:test s3 topics set/get/delete on master ... FAIL 2020-11-09T18:41:48.529 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:41:48.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:41:48.535 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:41:48.581 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:41:48.583 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:41:48.799 INFO:teuthology.orchestra.run.smithi051.stderr:No --period given, using current period=f6258aba-300a-4d5d-ad93-a7c198a0e0a1 2020-11-09T18:41:48.838 INFO:teuthology.orchestra.run.smithi051.stdout:[ 2020-11-09T18:41:48.838 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:41:48.838 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604946340.774139_451.1", 2020-11-09T18:41:48.839 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "2020-11-09T18:25:40.774139Z" 2020-11-09T18:41:48.839 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:41:48.839 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:41:48.839 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:41:48.840 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:41:48.842 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:41:48.843 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:41:48.843 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:41:48.843 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:41:48.844 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:41:48.844 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:41:48.845 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:41:48.845 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:41:48.846 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:41:48.847 INFO:teuthology.orchestra.run.smithi051.stdout:] 2020-11-09T18:41:48.849 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-09T18:41:48.852 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:41:48.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:41:48.854 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:41:48.901 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:41:48.902 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:41:49.165 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:41:49.165 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:41:49.165 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:41:49.166 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:41:49.166 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:41:49.166 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:41:49.166 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:41:49.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:49.166 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:41:49.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:49.167 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:41:49.167 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:49.167 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:49.167 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946340.774139_451.1", 2020-11-09T18:41:49.167 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:49.167 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:41:49.167 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:49.168 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:40.774139Z", 2020-11-09T18:41:49.168 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:41:49.168 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:49.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:49.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:49.168 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:41:49.168 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:49.169 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:49.169 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:41:49.169 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:49.169 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:41:49.169 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:49.169 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:41:49.170 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:41:49.170 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:49.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:49.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:49.170 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:41:49.170 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:49.171 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:49.171 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:41:49.171 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:49.171 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:41:49.171 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:49.172 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:41:49.172 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:41:49.172 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:49.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:49.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:49.173 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:41:49.173 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:49.173 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:49.173 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:41:49.173 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:49.173 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:41:49.174 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:49.174 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:41:49.174 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:41:49.174 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:49.174 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:49.174 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:41:49.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:49.175 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:41:49.175 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:41:49.175 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:41:49.175 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:49.175 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:41:49.176 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-09T18:41:49.177 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-09T18:41:49.177 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:41:49.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:41:49.178 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:41:49.222 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:41:49.222 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:41:49.479 INFO:teuthology.orchestra.run.smithi051.stdout:{ 2020-11-09T18:41:49.480 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_status": { 2020-11-09T18:41:49.480 INFO:teuthology.orchestra.run.smithi051.stdout: "info": { 2020-11-09T18:41:49.480 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "sync", 2020-11-09T18:41:49.480 INFO:teuthology.orchestra.run.smithi051.stdout: "num_shards": 4, 2020-11-09T18:41:49.480 INFO:teuthology.orchestra.run.smithi051.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:41:49.480 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 4 2020-11-09T18:41:49.481 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:41:49.481 INFO:teuthology.orchestra.run.smithi051.stdout: "markers": [ 2020-11-09T18:41:49.481 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:41:49.481 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 0, 2020-11-09T18:41:49.481 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:41:49.481 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:41:49.481 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604946340.774139_451.1", 2020-11-09T18:41:49.482 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:41:49.482 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 1, 2020-11-09T18:41:49.482 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:41:49.482 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:25:40.774139Z", 2020-11-09T18:41:49.482 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 4 2020-11-09T18:41:49.482 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:41:49.483 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:41:49.483 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:41:49.485 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 1, 2020-11-09T18:41:49.485 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:41:49.485 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:41:49.486 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:41:49.486 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:41:49.486 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:41:49.486 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:41:49.486 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:41:49.486 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:41:49.487 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:41:49.487 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:41:49.487 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:41:49.487 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 2, 2020-11-09T18:41:49.487 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:41:49.487 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:41:49.487 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:41:49.488 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:41:49.488 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:41:49.488 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:41:49.488 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:41:49.488 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:41:49.488 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:41:49.489 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:41:49.489 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:41:49.489 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 3, 2020-11-09T18:41:49.489 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:41:49.489 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:41:49.489 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:41:49.490 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:41:49.490 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:41:49.490 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:41:49.490 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:41:49.490 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:41:49.490 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:41:49.491 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:41:49.491 INFO:teuthology.orchestra.run.smithi051.stdout: ] 2020-11-09T18:41:49.491 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:41:49.491 INFO:teuthology.orchestra.run.smithi051.stdout: "full_sync": { 2020-11-09T18:41:49.492 INFO:teuthology.orchestra.run.smithi051.stdout: "total": 1, 2020-11-09T18:41:49.492 INFO:teuthology.orchestra.run.smithi051.stdout: "complete": 1 2020-11-09T18:41:49.492 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:41:49.492 INFO:teuthology.orchestra.run.smithi051.stdout:} 2020-11-09T18:41:49.493 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-09T18:41:49.494 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-09T18:41:49.494 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:41:49.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:41:49.495 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:41:49.535 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:41:49.536 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:41:49.821 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:41:49.822 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:41:49.822 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:41:49.822 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:41:49.822 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:41:49.823 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:41:49.823 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:41:49.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:49.823 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:41:49.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:49.824 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:41:49.824 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:49.824 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:49.824 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:41:49.824 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:49.825 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:41:49.825 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:49.825 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:41:49.825 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:41:49.825 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:49.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:49.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:49.826 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:41:49.826 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:49.826 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:49.826 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:41:49.826 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:49.827 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:41:49.827 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:49.827 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:41:49.827 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:41:49.827 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:49.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:49.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:49.828 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:41:49.828 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:49.828 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:49.829 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:41:49.829 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:49.829 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:41:49.829 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:49.829 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:41:49.830 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:41:49.830 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:49.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:49.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:49.830 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:41:49.830 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:49.831 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:49.831 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:41:49.831 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:49.831 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:41:49.831 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:49.831 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:41:49.832 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:41:49.832 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:49.832 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:49.832 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:41:49.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:49.833 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:41:49.833 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:41:49.833 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:41:49.833 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:49.834 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:41:49.834 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:41:54.835 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:41:54.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:41:54.837 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:41:54.878 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:41:54.879 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:41:55.144 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:41:55.145 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:41:55.146 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:41:55.146 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:41:55.146 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:41:55.146 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:41:55.146 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:41:55.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:55.146 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:41:55.147 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:55.147 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:41:55.147 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:55.147 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:55.147 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:41:55.147 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:55.147 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:41:55.148 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:55.148 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:41:55.148 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:41:55.148 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:55.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:55.148 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:55.148 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:41:55.149 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:55.149 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:55.149 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:41:55.149 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:55.149 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:41:55.149 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:55.149 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:41:55.150 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:41:55.150 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:55.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:55.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:55.150 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:41:55.150 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:55.150 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:55.151 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:41:55.151 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:55.151 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:41:55.151 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:55.151 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:41:55.151 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:41:55.151 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:55.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:55.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:41:55.152 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:41:55.152 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:41:55.152 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:41:55.152 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:41:55.152 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:41:55.152 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:41:55.153 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:41:55.153 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:41:55.153 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:41:55.153 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:55.153 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:55.153 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:41:55.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:41:55.153 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:41:55.154 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:41:55.154 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:41:55.154 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:41:55.154 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:41:55.155 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:42:00.157 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:42:00.158 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:42:00.158 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:42:00.201 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:42:00.202 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:42:00.568 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:42:00.568 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:42:00.569 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:42:00.569 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:42:00.569 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:42:00.569 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:42:00.569 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:42:00.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:00.570 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:42:00.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:42:00.571 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:42:00.571 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:42:00.571 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:42:00.571 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:42:00.571 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:42:00.571 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:42:00.572 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:42:00.572 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:42:00.572 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:42:00.572 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:00.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:00.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:42:00.574 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:42:00.574 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:42:00.574 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:42:00.574 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:42:00.574 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:42:00.574 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:42:00.575 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:42:00.575 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:42:00.575 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:42:00.575 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:00.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:00.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:42:00.575 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:42:00.575 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:42:00.576 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:42:00.576 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:42:00.576 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:42:00.576 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:42:00.576 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:42:00.576 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:42:00.576 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:42:00.577 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:00.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:00.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:42:00.577 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:42:00.577 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:42:00.577 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:42:00.577 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:42:00.578 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:42:00.578 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:42:00.578 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:42:00.578 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:42:00.578 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:42:00.578 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:00.578 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:00.578 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:42:00.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:00.579 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:42:00.579 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:42:00.579 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:42:00.579 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:00.579 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:42:00.580 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:42:05.581 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:42:05.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:42:05.586 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:42:05.633 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:42:05.633 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:42:05.990 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:42:05.990 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:42:05.990 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:42:05.991 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:42:05.991 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:42:05.991 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:42:05.991 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:42:05.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:05.991 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:42:05.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:42:05.992 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:42:05.992 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:42:05.992 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:42:05.992 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:42:05.992 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:42:05.992 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:42:05.992 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:42:05.993 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:42:05.993 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:42:05.993 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:05.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:05.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:42:05.993 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:42:05.993 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:42:05.994 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:42:05.994 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:42:05.994 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:42:05.994 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:42:05.994 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:42:05.994 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:42:05.994 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:42:05.995 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:05.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:05.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:42:05.996 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:42:05.996 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:42:05.996 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:42:05.996 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:42:05.997 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:42:05.997 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:42:05.997 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:42:05.997 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:42:05.998 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:42:05.998 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:05.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:05.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:42:05.999 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:42:05.999 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:42:06.000 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:42:06.000 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:42:06.000 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:42:06.000 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:42:06.001 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:42:06.001 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:42:06.001 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:42:06.001 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:06.002 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:06.002 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:42:06.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:06.003 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:42:06.003 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:42:06.003 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:42:06.004 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:06.004 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:42:06.005 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:42:11.007 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:42:11.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:42:11.008 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:42:11.054 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:42:11.054 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:42:11.420 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:42:11.420 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:42:11.420 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:42:11.421 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:42:11.421 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:42:11.421 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:42:11.421 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:42:11.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:11.421 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:42:11.421 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:42:11.422 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:42:11.422 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:42:11.422 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:42:11.422 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:42:11.422 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:42:11.422 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:42:11.422 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:42:11.423 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:42:11.423 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:42:11.423 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:11.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:11.423 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:42:11.423 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:42:11.423 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:42:11.424 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:42:11.424 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:42:11.424 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:42:11.424 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:42:11.424 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:42:11.424 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:42:11.425 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:42:11.425 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:11.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:11.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:42:11.425 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:42:11.425 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:42:11.426 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:42:11.426 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:42:11.426 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:42:11.426 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:42:11.426 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:42:11.426 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:42:11.427 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:42:11.427 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:11.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:11.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:42:11.427 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:42:11.427 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:42:11.427 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:42:11.428 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:42:11.428 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:42:11.428 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:42:11.428 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:42:11.428 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:42:11.428 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:42:11.429 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:11.429 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:11.429 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:42:11.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:11.429 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:42:11.429 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:42:11.430 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:42:11.430 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:11.430 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:42:11.431 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:42:16.432 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:42:16.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:42:16.434 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:42:16.475 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:42:16.476 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:42:16.848 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:42:16.849 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:42:16.849 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:42:16.849 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:42:16.850 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:42:16.850 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:42:16.850 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:42:16.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:16.850 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:42:16.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:42:16.851 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:42:16.851 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:42:16.851 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:42:16.851 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:42:16.851 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:42:16.851 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:42:16.852 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:42:16.852 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:42:16.852 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:42:16.852 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:16.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:16.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:42:16.853 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:42:16.853 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:42:16.853 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:42:16.853 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:42:16.853 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:42:16.853 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:42:16.854 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:42:16.854 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:42:16.854 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:42:16.854 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:16.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:16.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:42:16.855 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:42:16.855 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:42:16.855 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:42:16.855 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:42:16.855 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:42:16.855 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:42:16.856 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:42:16.856 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:42:16.856 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:42:16.856 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:16.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:16.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:42:16.857 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:42:16.857 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:42:16.857 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:42:16.857 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:42:16.857 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:42:16.857 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:42:16.857 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:42:16.858 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:42:16.858 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:42:16.858 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:16.858 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:16.858 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:42:16.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:16.859 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:42:16.859 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:42:16.859 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:42:16.859 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:16.860 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:42:16.861 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:42:21.862 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:42:21.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:42:21.864 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:42:21.905 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:42:21.905 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:42:22.273 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:42:22.273 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:42:22.273 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:42:22.274 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:42:22.274 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:42:22.274 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:42:22.274 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:42:22.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:22.274 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:42:22.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:42:22.275 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:42:22.275 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:42:22.275 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:42:22.275 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:42:22.275 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:42:22.275 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:42:22.275 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:42:22.276 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:42:22.276 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:42:22.276 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:22.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:22.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:42:22.276 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:42:22.276 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:42:22.277 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:42:22.277 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:42:22.277 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:42:22.277 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:42:22.277 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:42:22.277 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:42:22.278 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:42:22.278 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:22.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:22.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:42:22.278 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:42:22.278 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:42:22.278 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:42:22.279 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:42:22.279 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:42:22.279 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:42:22.279 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:42:22.279 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:42:22.279 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:42:22.279 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:22.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:22.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:42:22.280 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:42:22.280 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:42:22.280 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:42:22.280 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:42:22.280 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:42:22.280 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:42:22.281 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:42:22.281 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:42:22.281 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:42:22.281 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:22.281 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:22.281 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:42:22.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:22.281 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:42:22.282 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:42:22.282 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:42:22.282 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:22.282 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:42:22.283 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:42:27.285 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:42:27.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:42:27.287 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:42:27.328 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:42:27.329 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:42:27.697 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:42:27.698 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:42:27.698 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:42:27.698 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:42:27.698 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:42:27.699 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:42:27.699 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:42:27.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:27.699 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:42:27.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:42:27.699 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:42:27.700 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:42:27.700 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:42:27.700 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:42:27.701 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:42:27.701 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:42:27.701 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:42:27.701 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:42:27.702 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:42:27.702 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:27.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:27.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:42:27.702 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:42:27.702 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:42:27.703 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:42:27.703 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:42:27.703 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:42:27.703 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:42:27.703 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:42:27.703 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:42:27.703 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:42:27.703 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:27.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:27.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:42:27.704 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:42:27.704 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:42:27.704 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:42:27.704 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:42:27.704 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:42:27.705 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:42:27.705 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:42:27.705 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:42:27.705 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:42:27.706 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:27.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:27.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:42:27.706 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:42:27.707 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:42:27.707 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:42:27.707 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:42:27.707 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:42:27.707 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:42:27.707 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:42:27.707 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:42:27.708 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:42:27.708 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:27.708 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:27.708 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:42:27.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:27.708 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:42:27.708 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:42:27.709 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:42:27.709 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:27.709 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:42:27.710 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:42:32.711 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:42:32.712 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:42:32.712 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:42:32.757 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:42:32.758 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:42:33.116 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:42:33.116 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:42:33.116 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:42:33.116 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:42:33.117 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:42:33.117 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:42:33.117 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:42:33.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:33.117 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:42:33.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:42:33.118 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:42:33.118 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:42:33.118 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:42:33.119 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:42:33.119 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:42:33.119 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:42:33.119 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:42:33.119 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:42:33.119 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:42:33.119 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:33.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:33.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:42:33.120 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:42:33.120 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:42:33.120 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:42:33.120 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:42:33.120 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:42:33.121 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:42:33.121 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:42:33.121 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:42:33.121 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:42:33.121 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:33.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:33.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:42:33.121 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:42:33.122 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:42:33.122 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:42:33.122 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:42:33.122 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:42:33.122 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:42:33.122 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:42:33.122 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:42:33.123 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:42:33.123 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:33.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:33.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:42:33.123 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:42:33.123 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:42:33.123 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:42:33.124 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:42:33.124 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:42:33.124 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:42:33.124 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:42:33.124 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:42:33.124 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:42:33.124 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:33.124 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:33.125 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:42:33.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:33.125 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:42:33.125 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:42:33.125 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:42:33.125 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:33.125 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:42:33.126 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:42:38.127 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:42:38.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:42:38.129 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:42:38.172 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:42:38.173 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:42:38.542 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:42:38.542 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:42:38.542 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:42:38.543 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:42:38.543 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:42:38.543 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:42:38.543 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:42:38.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:38.543 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:42:38.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:42:38.544 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:42:38.544 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:42:38.544 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:42:38.544 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:42:38.544 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:42:38.544 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:42:38.544 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:42:38.545 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:42:38.545 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:42:38.545 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:38.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:38.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:42:38.545 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:42:38.545 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:42:38.546 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:42:38.546 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:42:38.546 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:42:38.546 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:42:38.546 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:42:38.546 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:42:38.546 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:42:38.547 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:38.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:38.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:42:38.547 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:42:38.547 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:42:38.547 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:42:38.547 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:42:38.547 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:42:38.548 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:42:38.548 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:42:38.548 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:42:38.548 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:42:38.548 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:38.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:38.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:42:38.549 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:42:38.549 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:42:38.549 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:42:38.549 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:42:38.549 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:42:38.549 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:42:38.549 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:42:38.549 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:42:38.550 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:42:38.550 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:38.550 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:38.550 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:42:38.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:38.550 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:42:38.550 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:42:38.551 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:42:38.551 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:38.551 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:42:38.552 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:42:43.553 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:42:43.554 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:42:43.554 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:42:43.595 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:42:43.596 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:42:43.948 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:42:43.948 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:42:43.949 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:42:43.949 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:42:43.949 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:42:43.949 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:42:43.950 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:42:43.950 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:43.950 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:42:43.950 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:42:43.950 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:42:43.950 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:42:43.951 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:42:43.951 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:42:43.951 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:42:43.951 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:42:43.951 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:42:43.951 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:42:43.951 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:42:43.952 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:43.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:43.952 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:42:43.953 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:42:43.953 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:42:43.953 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:42:43.953 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:42:43.953 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:42:43.953 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:42:43.954 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:42:43.954 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:42:43.954 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:42:43.954 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:43.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:43.954 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:42:43.954 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:42:43.955 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:42:43.955 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:42:43.955 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:42:43.955 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:42:43.955 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:42:43.955 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:42:43.955 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:42:43.956 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:42:43.956 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:43.956 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:43.956 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:42:43.956 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:42:43.956 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:42:43.956 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:42:43.957 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:42:43.957 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:42:43.957 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:42:43.957 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:42:43.957 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:42:43.957 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:42:43.957 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:43.958 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:43.958 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:42:43.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:43.958 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:42:43.958 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:42:43.958 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:42:43.958 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:43.959 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:42:43.960 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:42:48.962 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:42:48.963 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:42:48.963 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:42:49.003 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:42:49.004 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:42:49.373 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:42:49.374 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:42:49.374 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:42:49.374 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:42:49.374 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:42:49.374 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:42:49.374 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:42:49.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:49.375 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:42:49.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:42:49.375 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:42:49.375 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:42:49.375 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:42:49.375 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:42:49.376 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:42:49.376 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:42:49.376 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:42:49.376 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:42:49.376 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:42:49.376 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:49.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:49.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:42:49.377 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:42:49.377 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:42:49.377 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:42:49.377 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:42:49.377 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:42:49.377 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:42:49.377 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:42:49.378 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:42:49.378 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:42:49.378 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:49.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:49.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:42:49.378 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:42:49.379 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:42:49.379 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:42:49.379 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:42:49.379 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:42:49.379 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:42:49.379 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:42:49.379 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:42:49.379 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:42:49.380 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:49.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:49.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:42:49.380 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:42:49.380 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:42:49.380 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:42:49.380 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:42:49.381 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:42:49.381 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:42:49.381 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:42:49.381 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:42:49.381 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:42:49.381 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:49.381 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:49.381 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:42:49.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:49.382 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:42:49.382 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:42:49.382 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:42:49.382 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:49.382 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:42:49.383 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:42:54.385 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:42:54.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:42:54.387 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:42:54.429 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:42:54.429 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:42:54.694 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:42:54.694 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:42:54.696 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:42:54.696 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:42:54.696 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:42:54.697 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:42:54.697 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:42:54.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:54.697 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:42:54.697 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:42:54.698 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:42:54.698 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:42:54.699 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:42:54.699 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:42:54.699 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:42:54.699 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:42:54.699 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:42:54.700 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:42:54.700 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:42:54.700 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:54.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:54.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:42:54.700 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:42:54.700 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:42:54.701 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:42:54.701 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:42:54.701 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:42:54.701 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:42:54.701 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:42:54.701 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:42:54.702 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:42:54.702 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:54.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:54.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:42:54.702 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:42:54.702 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:42:54.702 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:42:54.703 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:42:54.703 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:42:54.703 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:42:54.703 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:42:54.703 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:42:54.703 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:42:54.704 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:54.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:54.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:42:54.704 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:42:54.704 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:42:54.704 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:42:54.704 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:42:54.705 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:42:54.705 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:42:54.705 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:42:54.705 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:42:54.705 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:42:54.705 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:54.706 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:54.706 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:42:54.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:42:54.706 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:42:54.706 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:42:54.706 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:42:54.706 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:42:54.707 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:42:54.708 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:42:59.709 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:42:59.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:42:59.710 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:42:59.751 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:42:59.751 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:43:00.124 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:43:00.125 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:43:00.125 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:43:00.125 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:43:00.125 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:43:00.125 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:43:00.126 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:43:00.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:00.126 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:43:00.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:00.126 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:43:00.126 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:00.127 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:00.127 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:43:00.127 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:00.127 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:43:00.127 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:00.127 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:43:00.128 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:43:00.128 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:00.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:00.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:00.129 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:43:00.129 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:00.129 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:00.129 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:43:00.129 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:00.129 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:43:00.130 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:00.130 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:43:00.130 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:43:00.130 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:00.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:00.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:00.130 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:43:00.131 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:00.132 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:00.132 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:43:00.132 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:00.132 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:43:00.132 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:00.133 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:43:00.133 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:43:00.133 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:00.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:00.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:00.133 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:43:00.133 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:00.134 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:00.134 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:43:00.134 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:00.134 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:43:00.134 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:00.134 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:43:00.134 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:43:00.135 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:00.135 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:00.135 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:43:00.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:00.135 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:43:00.135 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:43:00.135 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:43:00.135 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:00.136 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:43:00.137 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:43:05.138 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:43:05.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:43:05.140 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:43:05.184 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:43:05.184 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:43:05.555 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:43:05.556 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:43:05.556 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:43:05.556 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:43:05.556 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:43:05.556 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:43:05.557 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:43:05.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:05.557 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:43:05.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:05.557 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:43:05.557 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:05.557 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:05.558 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:43:05.558 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:05.558 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:43:05.558 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:05.558 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:43:05.558 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:43:05.559 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:05.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:05.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:05.559 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:43:05.559 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:05.559 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:05.559 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:43:05.560 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:05.560 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:43:05.560 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:05.560 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:43:05.560 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:43:05.560 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:05.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:05.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:05.561 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:43:05.561 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:05.561 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:05.561 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:43:05.561 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:05.562 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:43:05.562 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:05.562 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:43:05.562 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:43:05.563 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:05.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:05.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:05.563 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:43:05.565 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:05.566 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:05.566 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:43:05.566 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:05.566 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:43:05.567 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:05.567 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:43:05.567 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:43:05.567 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:05.567 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:05.567 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:43:05.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:05.568 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:43:05.568 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:43:05.568 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:43:05.568 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:05.568 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:43:05.569 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:43:10.571 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:43:10.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:43:10.572 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:43:10.608 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:43:10.609 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:43:10.967 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:43:10.968 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:43:10.968 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:43:10.968 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:43:10.968 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:43:10.968 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:43:10.968 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:43:10.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:10.969 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:43:10.969 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:10.969 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:43:10.970 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:10.970 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:10.970 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:43:10.970 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:10.970 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:43:10.970 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:10.971 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:43:10.971 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:43:10.971 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:10.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:10.972 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:10.972 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:43:10.972 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:10.972 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:10.972 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:43:10.972 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:10.972 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:43:10.972 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:10.973 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:43:10.973 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:43:10.973 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:10.973 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:10.973 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:10.973 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:43:10.973 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:10.974 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:10.974 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:43:10.974 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:10.974 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:43:10.974 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:10.974 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:43:10.974 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:43:10.975 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:10.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:10.975 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:10.975 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:43:10.975 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:10.975 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:10.975 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:43:10.975 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:10.976 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:43:10.976 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:10.976 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:43:10.976 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:43:10.976 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:10.976 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:10.976 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:43:10.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:10.977 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:43:10.977 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:43:10.977 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:43:10.977 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:10.977 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:43:10.978 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:43:15.980 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:43:15.981 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:43:15.982 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:43:16.025 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:43:16.025 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:43:16.393 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:43:16.394 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:43:16.394 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:43:16.395 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:43:16.395 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:43:16.395 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:43:16.396 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:43:16.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:16.396 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:43:16.396 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:16.397 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:43:16.397 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:16.397 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:16.398 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:43:16.398 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:16.398 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:43:16.399 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:16.399 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:43:16.399 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:43:16.399 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:16.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:16.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:16.400 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:43:16.401 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:16.401 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:16.401 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:43:16.401 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:16.402 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:43:16.402 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:16.402 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:43:16.403 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:43:16.403 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:16.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:16.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:16.404 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:43:16.404 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:16.404 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:16.405 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:43:16.405 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:16.405 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:43:16.405 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:16.406 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:43:16.406 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:43:16.406 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:16.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:16.407 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:16.407 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:43:16.407 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:16.408 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:16.408 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:43:16.408 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:16.409 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:43:16.409 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:16.409 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:43:16.409 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:43:16.410 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:16.410 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:16.410 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:43:16.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:16.411 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:43:16.411 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:43:16.411 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:43:16.412 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:16.412 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:43:16.414 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:43:21.416 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:43:21.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:43:21.417 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:43:21.459 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:43:21.459 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:43:21.829 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:43:21.829 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:43:21.830 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:43:21.830 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:43:21.831 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:43:21.831 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:43:21.831 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:43:21.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:21.832 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:43:21.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:21.833 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:43:21.833 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:21.833 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:21.834 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:43:21.834 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:21.834 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:43:21.834 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:21.835 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:43:21.835 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:43:21.835 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:21.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:21.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:21.836 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:43:21.836 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:21.837 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:21.837 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:43:21.837 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:21.838 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:43:21.838 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:21.838 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:43:21.839 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:43:21.839 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:21.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:21.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:21.840 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:43:21.840 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:21.840 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:21.840 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:43:21.841 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:21.841 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:43:21.841 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:21.841 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:43:21.841 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:43:21.842 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:21.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:21.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:21.842 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:43:21.843 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:21.843 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:21.843 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:43:21.844 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:21.844 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:43:21.844 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:21.844 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:43:21.844 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:43:21.845 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:21.845 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:21.845 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:43:21.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:21.846 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:43:21.846 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:43:21.846 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:43:21.846 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:21.847 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:43:21.849 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:43:26.851 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:43:26.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:43:26.853 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:43:26.893 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:43:26.894 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:43:27.258 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:43:27.259 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:43:27.259 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:43:27.259 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:43:27.259 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:43:27.259 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:43:27.259 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:43:27.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:27.260 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:43:27.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:27.260 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:43:27.260 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:27.260 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:27.260 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:43:27.261 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:27.261 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:43:27.261 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:27.261 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:43:27.261 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:43:27.261 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:27.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:27.262 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:27.262 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:43:27.262 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:27.262 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:27.262 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:43:27.262 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:27.263 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:43:27.263 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:27.263 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:43:27.263 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:43:27.263 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:27.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:27.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:27.263 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:43:27.264 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:27.264 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:27.264 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:43:27.264 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:27.264 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:43:27.264 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:27.264 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:43:27.265 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:43:27.265 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:27.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:27.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:27.265 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:43:27.265 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:27.265 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:27.265 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:43:27.266 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:27.266 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:43:27.266 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:27.266 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:43:27.266 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:43:27.266 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:27.266 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:27.267 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:43:27.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:27.267 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:43:27.267 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:43:27.267 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:43:27.267 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:27.267 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:43:27.268 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:43:32.270 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:43:32.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:43:32.272 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:43:32.316 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:43:32.317 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:43:32.689 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:43:32.689 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:43:32.689 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:43:32.689 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:43:32.690 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:43:32.690 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:43:32.690 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:43:32.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:32.690 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:43:32.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:32.690 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:43:32.691 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:32.691 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:32.691 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:43:32.691 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:32.691 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:43:32.691 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:32.691 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:43:32.692 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:43:32.692 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:32.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:32.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:32.699 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:43:32.699 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:32.699 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:32.700 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:43:32.700 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:32.700 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:43:32.700 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:32.700 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:43:32.700 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:43:32.701 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:32.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:32.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:32.701 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:43:32.701 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:32.701 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:32.701 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:43:32.702 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:32.702 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:43:32.702 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:32.702 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:43:32.702 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:43:32.702 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:32.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:32.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:32.703 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:43:32.703 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:32.703 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:32.703 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:43:32.703 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:32.703 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:43:32.704 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:32.704 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:43:32.704 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:43:32.704 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:32.704 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:32.704 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:43:32.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:32.705 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:43:32.705 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:43:32.705 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:43:32.705 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:32.705 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:43:32.706 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:43:37.706 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:43:37.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:43:37.708 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:43:37.749 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:43:37.749 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:43:38.121 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:43:38.121 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:43:38.121 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:43:38.121 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:43:38.121 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:43:38.122 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:43:38.122 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:43:38.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:38.122 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:43:38.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:38.122 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:43:38.122 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:38.123 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:38.123 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:43:38.123 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:38.123 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:43:38.123 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:38.123 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:43:38.123 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:43:38.124 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:38.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:38.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:38.124 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:43:38.124 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:38.124 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:38.124 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:43:38.125 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:38.125 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:43:38.125 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:38.125 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:43:38.125 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:43:38.125 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:38.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:38.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:38.126 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:43:38.126 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:38.126 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:38.126 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:43:38.126 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:38.126 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:43:38.126 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:38.127 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:43:38.127 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:43:38.127 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:38.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:38.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:38.127 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:43:38.127 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:38.127 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:38.128 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:43:38.128 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:38.128 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:43:38.128 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:38.128 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:43:38.128 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:43:38.128 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:38.129 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:38.129 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:43:38.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:38.129 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:43:38.129 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:43:38.129 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:43:38.129 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:38.129 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:43:38.131 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:43:43.133 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:43:43.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:43:43.135 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:43:43.175 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:43:43.176 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:43:43.545 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:43:43.546 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:43:43.546 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:43:43.546 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:43:43.546 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:43:43.546 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:43:43.546 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:43:43.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:43.547 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:43:43.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:43.547 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:43:43.547 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:43.547 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:43.547 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:43:43.547 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:43.548 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:43:43.548 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:43.548 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:43:43.548 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:43:43.548 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:43.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:43.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:43.549 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:43:43.549 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:43.549 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:43.549 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:43:43.549 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:43.549 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:43:43.549 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:43.549 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:43:43.550 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:43:43.550 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:43.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:43.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:43.550 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:43:43.550 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:43.551 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:43.551 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:43:43.551 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:43.551 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:43:43.551 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:43.551 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:43:43.551 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:43:43.551 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:43.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:43.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:43.552 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:43:43.552 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:43.552 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:43.553 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:43:43.553 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:43.553 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:43:43.553 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:43.553 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:43:43.554 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:43:43.554 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:43.554 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:43.554 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:43:43.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:43.554 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:43:43.555 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:43:43.555 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:43:43.555 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:43.555 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:43:43.556 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:43:48.557 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:43:48.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:43:48.566 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:43:48.614 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:43:48.615 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:43:48.883 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:43:48.885 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:43:48.885 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:43:48.885 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:43:48.885 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:43:48.885 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:43:48.886 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:43:48.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:48.886 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:43:48.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:48.886 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:43:48.886 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:48.887 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:48.887 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:43:48.887 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:48.887 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:43:48.887 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:48.887 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:43:48.888 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:43:48.888 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:48.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:48.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:48.888 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:43:48.888 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:48.888 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:48.889 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:43:48.889 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:48.889 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:43:48.889 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:48.889 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:43:48.889 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:43:48.890 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:48.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:48.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:48.890 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:43:48.890 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:48.891 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:48.891 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:43:48.891 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:48.891 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:43:48.891 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:48.891 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:43:48.891 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:43:48.892 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:48.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:48.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:48.892 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:43:48.892 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:48.893 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:48.893 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:43:48.893 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:48.893 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:43:48.893 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:48.893 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:43:48.894 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:43:48.894 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:48.894 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:48.894 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:43:48.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:48.894 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:43:48.895 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:43:48.895 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:43:48.895 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:48.895 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:43:48.896 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:43:53.898 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:43:53.898 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:43:53.899 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:43:53.941 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:43:53.941 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:43:54.305 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:43:54.305 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:43:54.305 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:43:54.306 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:43:54.306 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:43:54.306 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:43:54.306 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:43:54.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:54.306 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:43:54.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:54.307 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:43:54.307 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:54.307 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:54.307 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:43:54.307 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:54.307 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:43:54.308 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:54.308 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:43:54.308 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:43:54.308 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:54.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:54.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:54.309 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:43:54.309 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:54.309 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:54.309 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:43:54.309 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:54.310 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:43:54.310 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:54.310 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:43:54.310 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:43:54.310 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:54.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:54.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:54.311 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:43:54.311 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:54.311 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:54.312 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:43:54.312 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:54.312 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:43:54.312 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:54.312 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:43:54.313 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:43:54.313 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:54.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:54.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:54.313 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:43:54.313 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:54.313 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:54.313 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:43:54.314 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:54.314 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:43:54.314 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:54.314 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:43:54.314 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:43:54.314 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:54.314 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:54.315 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:43:54.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:54.315 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:43:54.315 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:43:54.315 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:43:54.315 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:54.315 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:43:54.316 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:43:59.318 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:43:59.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:43:59.319 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:43:59.361 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:43:59.361 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:43:59.732 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:43:59.732 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:43:59.732 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:43:59.732 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:43:59.733 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:43:59.733 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:43:59.733 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:43:59.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:59.733 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:43:59.733 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:59.733 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:43:59.734 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:59.734 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:59.734 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:43:59.734 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:59.734 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:43:59.734 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:59.734 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:43:59.735 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:43:59.735 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:59.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:59.735 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:59.735 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:43:59.735 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:59.736 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:59.736 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:43:59.736 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:59.736 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:43:59.736 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:59.736 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:43:59.736 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:43:59.737 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:59.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:59.737 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:59.737 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:43:59.737 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:59.738 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:59.738 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:43:59.738 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:59.738 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:43:59.738 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:59.738 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:43:59.738 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:43:59.738 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:59.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:59.739 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:43:59.739 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:43:59.739 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:43:59.739 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:43:59.739 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:43:59.739 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:43:59.740 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:43:59.740 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:43:59.740 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:43:59.740 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:43:59.740 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:59.740 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:59.740 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:43:59.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:43:59.741 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:43:59.741 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:43:59.741 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:43:59.741 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:43:59.741 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:43:59.743 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:44:04.744 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:44:04.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:44:04.752 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:44:04.809 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:44:04.809 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:44:05.171 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:44:05.171 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:44:05.171 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:44:05.171 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:44:05.172 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:44:05.172 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:44:05.172 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:44:05.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:05.172 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:44:05.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:44:05.172 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:44:05.173 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:44:05.173 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:44:05.173 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:44:05.173 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:44:05.173 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:44:05.173 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:44:05.173 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:44:05.174 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:44:05.178 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:05.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:05.179 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:44:05.179 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:44:05.179 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:44:05.180 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:44:05.180 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:44:05.180 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:44:05.180 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:44:05.180 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:44:05.180 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:44:05.180 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:44:05.181 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:05.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:05.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:44:05.181 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:44:05.181 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:44:05.181 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:44:05.181 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:44:05.182 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:44:05.182 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:44:05.182 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:44:05.182 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:44:05.182 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:44:05.182 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:05.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:05.183 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:44:05.183 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:44:05.183 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:44:05.183 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:44:05.183 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:44:05.183 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:44:05.183 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:44:05.184 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:44:05.184 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:44:05.184 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:44:05.184 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:05.184 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:05.184 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:44:05.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:05.184 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:44:05.186 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:44:05.187 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:44:05.187 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:05.187 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:44:05.188 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:44:10.189 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:44:10.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:44:10.190 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:44:10.232 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:44:10.232 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:44:10.495 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:44:10.496 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:44:10.496 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:44:10.496 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:44:10.496 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:44:10.497 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:44:10.497 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:44:10.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:10.497 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:44:10.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:44:10.497 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:44:10.497 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:44:10.498 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:44:10.498 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:44:10.498 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:44:10.498 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:44:10.498 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:44:10.498 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:44:10.498 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:44:10.499 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:10.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:10.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:44:10.499 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:44:10.499 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:44:10.499 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:44:10.499 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:44:10.500 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:44:10.500 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:44:10.500 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:44:10.500 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:44:10.500 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:44:10.500 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:10.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:10.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:44:10.501 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:44:10.501 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:44:10.501 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:44:10.501 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:44:10.501 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:44:10.502 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:44:10.502 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:44:10.502 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:44:10.502 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:44:10.502 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:10.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:10.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:44:10.503 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:44:10.503 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:44:10.503 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:44:10.503 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:44:10.503 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:44:10.503 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:44:10.503 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:44:10.504 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:44:10.504 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:44:10.504 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:10.504 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:10.504 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:44:10.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:10.504 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:44:10.505 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:44:10.505 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:44:10.505 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:10.505 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:44:10.506 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:44:15.508 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:44:15.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:44:15.510 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:44:15.553 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:44:15.554 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:44:15.918 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:44:15.918 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:44:15.918 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:44:15.918 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:44:15.919 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:44:15.919 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:44:15.919 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:44:15.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:15.919 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:44:15.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:44:15.919 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:44:15.920 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:44:15.920 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:44:15.920 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:44:15.920 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:44:15.920 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:44:15.920 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:44:15.920 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:44:15.921 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:44:15.921 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:15.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:15.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:44:15.921 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:44:15.921 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:44:15.921 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:44:15.922 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:44:15.922 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:44:15.922 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:44:15.922 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:44:15.922 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:44:15.922 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:44:15.922 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:15.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:15.923 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:44:15.923 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:44:15.923 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:44:15.923 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:44:15.923 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:44:15.923 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:44:15.923 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:44:15.924 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:44:15.924 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:44:15.924 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:44:15.924 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:15.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:15.924 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:44:15.924 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:44:15.925 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:44:15.925 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:44:15.925 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:44:15.925 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:44:15.925 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:44:15.925 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:44:15.925 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:44:15.925 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:44:15.926 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:15.926 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:15.926 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:44:15.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:15.926 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:44:15.927 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:44:15.927 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:44:15.927 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:15.927 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:44:15.928 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:44:20.929 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:44:20.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:44:20.931 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:44:20.984 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:44:20.984 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:44:21.356 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:44:21.357 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:44:21.357 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:44:21.357 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:44:21.358 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:44:21.358 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:44:21.358 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:44:21.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:21.359 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:44:21.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:44:21.359 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:44:21.360 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:44:21.360 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:44:21.360 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:44:21.361 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:44:21.361 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:44:21.361 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:44:21.361 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:44:21.362 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:44:21.364 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:21.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:21.364 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:44:21.365 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:44:21.365 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:44:21.365 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:44:21.365 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:44:21.365 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:44:21.366 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:44:21.366 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:44:21.366 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:44:21.366 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:44:21.366 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:21.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:21.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:44:21.367 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:44:21.367 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:44:21.367 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:44:21.367 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:44:21.368 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:44:21.368 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:44:21.368 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:44:21.368 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:44:21.368 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:44:21.368 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:21.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:21.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:44:21.369 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:44:21.369 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:44:21.369 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:44:21.369 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:44:21.369 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:44:21.369 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:44:21.369 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:44:21.370 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:44:21.370 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:44:21.370 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:21.370 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:21.370 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:44:21.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:21.370 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:44:21.371 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:44:21.371 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:44:21.371 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:21.371 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:44:21.372 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:44:26.373 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:44:26.374 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:44:26.375 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:44:26.415 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:44:26.415 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:44:26.782 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:44:26.783 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:44:26.783 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:44:26.783 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:44:26.784 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:44:26.784 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:44:26.784 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:44:26.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:26.785 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:44:26.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:44:26.785 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:44:26.785 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:44:26.785 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:44:26.786 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:44:26.786 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:44:26.786 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:44:26.786 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:44:26.786 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:44:26.787 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:44:26.787 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:26.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:26.787 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:44:26.787 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:44:26.788 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:44:26.788 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:44:26.788 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:44:26.788 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:44:26.788 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:44:26.789 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:44:26.789 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:44:26.789 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:44:26.789 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:26.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:26.790 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:44:26.790 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:44:26.791 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:44:26.792 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:44:26.792 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:44:26.792 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:44:26.792 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:44:26.793 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:44:26.793 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:44:26.794 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:44:26.794 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:26.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:26.794 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:44:26.794 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:44:26.794 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:44:26.795 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:44:26.795 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:44:26.795 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:44:26.795 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:44:26.795 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:44:26.796 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:44:26.796 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:44:26.796 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:26.796 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:26.796 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:44:26.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:26.797 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:44:26.797 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:44:26.797 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:44:26.798 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:26.798 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:44:26.799 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:44:31.800 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:44:31.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:44:31.802 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:44:31.843 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:44:31.843 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:44:32.101 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:44:32.101 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:44:32.101 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:44:32.101 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:44:32.102 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:44:32.102 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:44:32.102 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:44:32.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:32.102 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:44:32.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:44:32.102 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:44:32.103 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:44:32.103 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:44:32.103 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:44:32.103 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:44:32.103 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:44:32.103 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:44:32.103 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:44:32.104 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:44:32.104 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:32.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:32.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:44:32.104 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:44:32.104 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:44:32.104 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:44:32.105 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:44:32.105 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:44:32.105 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:44:32.106 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:44:32.106 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:44:32.106 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:44:32.106 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:32.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:32.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:44:32.106 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:44:32.107 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:44:32.107 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:44:32.107 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:44:32.108 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:44:32.108 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:44:32.108 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:44:32.108 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:44:32.108 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:44:32.108 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:32.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:32.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:44:32.109 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:44:32.109 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:44:32.109 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:44:32.109 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:44:32.109 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:44:32.109 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:44:32.110 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:44:32.110 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:44:32.110 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:44:32.111 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:32.111 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:32.111 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:44:32.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:32.111 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:44:32.111 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:44:32.111 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:44:32.112 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:32.112 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:44:32.113 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:44:37.114 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:44:37.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:44:37.115 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:44:37.155 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:44:37.155 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:44:37.525 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:44:37.525 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:44:37.525 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:44:37.526 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:44:37.526 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:44:37.526 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:44:37.526 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:44:37.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:37.526 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:44:37.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:44:37.527 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:44:37.527 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:44:37.527 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:44:37.529 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:44:37.529 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:44:37.529 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:44:37.530 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:44:37.530 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:44:37.530 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:44:37.530 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:37.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:37.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:44:37.531 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:44:37.532 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:44:37.532 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:44:37.532 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:44:37.533 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:44:37.533 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:44:37.533 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:44:37.533 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:44:37.534 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:44:37.534 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:37.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:37.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:44:37.535 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:44:37.535 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:44:37.535 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:44:37.536 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:44:37.536 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:44:37.536 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:44:37.537 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:44:37.537 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:44:37.537 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:44:37.537 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:37.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:37.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:44:37.538 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:44:37.539 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:44:37.539 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:44:37.539 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:44:37.539 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:44:37.540 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:44:37.540 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:44:37.540 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:44:37.541 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:44:37.541 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:37.541 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:37.541 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:44:37.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:37.543 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:44:37.543 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:44:37.543 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:44:37.543 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:37.544 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:44:37.545 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:44:42.546 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:44:42.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:44:42.548 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:44:42.591 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:44:42.592 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:44:42.959 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:44:42.959 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:44:42.959 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:44:42.960 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:44:42.960 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:44:42.960 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:44:42.960 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:44:42.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:42.960 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:44:42.961 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:44:42.961 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:44:42.961 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:44:42.961 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:44:42.961 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:44:42.961 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:44:42.962 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:44:42.962 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:44:42.962 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:44:42.962 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:44:42.962 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:42.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:42.962 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:44:42.963 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:44:42.963 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:44:42.963 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:44:42.963 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:44:42.963 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:44:42.963 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:44:42.963 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:44:42.964 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:44:42.964 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:44:42.964 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:42.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:42.964 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:44:42.964 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:44:42.965 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:44:42.965 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:44:42.965 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:44:42.965 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:44:42.965 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:44:42.965 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:44:42.965 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:44:42.966 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:44:42.966 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:42.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:42.966 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:44:42.966 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:44:42.966 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:44:42.966 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:44:42.967 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:44:42.967 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:44:42.967 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:44:42.967 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:44:42.967 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:44:42.967 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:44:42.968 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:42.968 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:42.968 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:44:42.968 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:42.968 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:44:42.968 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:44:42.969 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:44:42.969 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:42.969 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:44:42.970 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:44:47.971 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:44:47.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:44:47.975 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:44:48.018 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:44:48.019 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:44:48.381 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:44:48.381 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:44:48.381 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:44:48.382 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:44:48.382 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:44:48.382 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:44:48.382 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:44:48.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:48.382 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:44:48.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:44:48.383 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:44:48.383 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:44:48.383 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:44:48.383 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:44:48.383 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:44:48.383 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:44:48.384 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:44:48.384 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:44:48.384 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:44:48.384 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:48.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:48.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:44:48.385 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:44:48.385 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:44:48.385 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:44:48.385 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:44:48.385 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:44:48.385 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:44:48.385 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:44:48.386 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:44:48.386 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:44:48.387 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:48.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:48.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:44:48.387 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:44:48.387 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:44:48.387 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:44:48.388 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:44:48.388 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:44:48.388 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:44:48.388 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:44:48.388 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:44:48.391 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:44:48.391 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:48.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:48.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:44:48.391 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:44:48.391 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:44:48.391 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:44:48.392 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:44:48.392 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:44:48.392 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:44:48.392 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:44:48.392 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:44:48.392 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:44:48.392 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:48.393 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:48.393 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:44:48.393 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:48.393 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:44:48.393 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:44:48.393 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:44:48.393 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:48.394 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:44:48.395 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:44:53.397 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:44:53.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:44:53.398 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:44:53.442 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:44:53.443 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:44:53.811 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:44:53.811 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:44:53.811 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:44:53.811 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:44:53.812 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:44:53.812 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:44:53.812 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:44:53.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:53.812 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:44:53.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:44:53.813 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:44:53.813 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:44:53.813 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:44:53.813 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:44:53.813 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:44:53.814 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:44:53.814 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:44:53.814 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:44:53.815 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:44:53.815 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:53.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:53.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:44:53.816 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:44:53.816 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:44:53.816 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:44:53.816 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:44:53.816 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:44:53.817 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:44:53.817 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:44:53.817 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:44:53.818 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:44:53.818 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:53.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:53.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:44:53.818 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:44:53.819 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:44:53.819 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:44:53.819 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:44:53.819 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:44:53.820 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:44:53.820 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:44:53.820 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:44:53.821 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:44:53.821 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:53.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:53.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:44:53.821 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:44:53.821 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:44:53.822 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:44:53.822 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:44:53.822 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:44:53.823 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:44:53.823 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:44:53.823 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:44:53.823 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:44:53.824 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:53.824 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:53.824 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:44:53.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:53.824 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:44:53.825 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:44:53.825 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:44:53.825 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:53.825 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:44:53.826 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:44:58.828 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:44:58.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:44:58.830 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:44:58.874 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:44:58.875 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:44:59.341 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:44:59.342 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:44:59.342 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:44:59.342 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:44:59.342 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:44:59.343 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:44:59.343 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:44:59.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:59.343 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:44:59.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:44:59.344 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:44:59.344 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:44:59.344 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:44:59.344 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:44:59.344 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:44:59.345 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:44:59.345 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:44:59.345 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:44:59.345 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:44:59.345 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:59.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:59.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:44:59.346 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:44:59.346 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:44:59.346 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:44:59.346 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:44:59.346 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:44:59.346 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:44:59.347 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:44:59.347 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:44:59.347 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:44:59.347 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:59.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:59.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:44:59.347 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:44:59.348 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:44:59.348 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:44:59.348 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:44:59.348 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:44:59.348 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:44:59.348 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:44:59.348 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:44:59.349 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:44:59.349 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:59.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:59.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:44:59.349 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:44:59.349 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:44:59.350 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:44:59.350 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:44:59.350 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:44:59.350 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:44:59.350 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:44:59.350 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:44:59.351 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:44:59.351 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:59.351 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:59.351 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:44:59.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:44:59.351 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:44:59.351 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:44:59.352 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:44:59.352 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:44:59.352 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:44:59.353 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:45:04.354 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:45:04.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:45:04.355 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:45:04.396 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:45:04.396 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:45:04.760 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:45:04.761 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:45:04.761 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:45:04.761 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:45:04.762 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:45:04.762 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:45:04.762 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:45:04.762 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:04.762 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:45:04.762 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:45:04.763 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:45:04.763 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:45:04.763 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:45:04.763 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:45:04.763 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:45:04.763 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:45:04.764 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:45:04.764 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:45:04.764 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:45:04.764 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:04.764 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:04.764 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:45:04.765 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:45:04.765 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:45:04.765 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:45:04.765 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:45:04.765 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:45:04.765 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:45:04.766 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:45:04.766 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:45:04.766 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:45:04.766 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:04.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:04.766 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:45:04.767 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:45:04.767 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:45:04.767 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:45:04.767 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:45:04.767 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:45:04.767 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:45:04.767 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:45:04.768 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:45:04.768 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:45:04.768 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:04.768 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:04.768 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:45:04.769 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:45:04.769 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:45:04.769 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:45:04.769 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:45:04.769 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:45:04.769 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:45:04.770 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:45:04.770 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:45:04.770 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:45:04.770 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:04.770 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:04.770 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:45:04.771 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:04.771 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:45:04.772 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:45:04.772 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:45:04.772 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:04.772 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:45:04.773 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:45:09.775 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:45:09.776 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:45:09.777 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:45:09.818 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:45:09.819 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:45:10.186 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:45:10.187 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:45:10.187 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:45:10.187 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:45:10.188 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:45:10.188 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:45:10.188 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:45:10.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:10.189 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:45:10.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:45:10.189 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:45:10.190 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:45:10.190 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:45:10.190 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:45:10.191 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:45:10.191 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:45:10.191 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:45:10.191 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:45:10.192 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:45:10.192 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:10.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:10.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:45:10.193 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:45:10.193 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:45:10.193 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:45:10.194 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:45:10.194 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:45:10.194 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:45:10.195 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:45:10.195 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:45:10.195 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:45:10.196 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:10.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:10.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:45:10.197 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:45:10.197 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:45:10.198 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:45:10.198 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:45:10.198 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:45:10.199 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:45:10.199 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:45:10.199 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:45:10.199 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:45:10.200 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:10.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:10.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:45:10.201 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:45:10.201 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:45:10.201 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:45:10.201 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:45:10.202 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:45:10.202 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:45:10.202 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:45:10.203 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:45:10.203 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:45:10.203 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:10.203 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:10.204 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:45:10.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:10.204 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:45:10.205 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:45:10.205 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:45:10.205 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:10.205 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:45:10.208 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:45:15.209 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:45:15.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:45:15.210 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:45:15.251 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:45:15.252 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:45:15.617 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:45:15.617 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:45:15.618 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:45:15.618 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:45:15.618 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:45:15.618 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:45:15.618 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:45:15.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:15.619 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:45:15.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:45:15.619 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:45:15.619 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:45:15.619 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:45:15.619 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:45:15.619 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:45:15.620 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:45:15.620 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:45:15.620 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:45:15.620 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:45:15.620 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:15.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:15.620 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:45:15.620 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:45:15.621 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:45:15.621 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:45:15.621 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:45:15.621 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:45:15.621 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:45:15.621 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:45:15.621 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:45:15.622 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:45:15.622 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:15.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:15.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:45:15.622 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:45:15.622 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:45:15.622 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:45:15.623 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:45:15.623 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:45:15.623 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:45:15.623 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:45:15.623 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:45:15.623 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:45:15.623 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:15.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:15.624 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:45:15.624 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:45:15.624 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:45:15.624 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:45:15.624 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:45:15.624 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:45:15.624 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:45:15.625 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:45:15.625 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:45:15.625 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:45:15.625 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:15.625 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:15.625 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:45:15.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:15.626 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:45:15.626 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:45:15.626 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:45:15.626 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:15.626 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:45:15.627 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:45:20.629 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:45:20.630 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:45:20.630 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:45:20.672 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:45:20.672 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:45:21.042 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:45:21.043 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:45:21.043 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:45:21.043 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:45:21.043 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:45:21.043 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:45:21.044 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:45:21.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:21.044 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:45:21.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:45:21.044 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:45:21.044 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:45:21.044 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:45:21.045 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:45:21.045 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:45:21.045 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:45:21.045 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:45:21.045 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:45:21.045 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:45:21.045 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:21.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:21.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:45:21.046 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:45:21.046 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:45:21.046 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:45:21.046 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:45:21.046 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:45:21.046 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:45:21.047 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:45:21.047 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:45:21.047 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:45:21.047 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:21.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:21.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:45:21.047 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:45:21.048 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:45:21.048 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:45:21.048 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:45:21.048 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:45:21.048 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:45:21.048 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:45:21.048 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:45:21.049 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:45:21.049 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:21.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:21.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:45:21.049 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:45:21.049 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:45:21.049 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:45:21.050 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:45:21.050 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:45:21.050 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:45:21.050 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:45:21.050 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:45:21.051 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:45:21.051 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:21.051 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:21.051 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:45:21.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:21.051 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:45:21.051 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:45:21.052 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:45:21.052 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:21.052 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:45:21.053 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:45:26.054 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:45:26.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:45:26.056 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:45:26.098 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:45:26.098 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:45:26.466 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:45:26.466 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:45:26.466 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:45:26.466 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:45:26.467 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:45:26.467 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:45:26.467 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:45:26.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:26.467 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:45:26.467 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:45:26.467 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:45:26.468 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:45:26.468 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:45:26.468 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:45:26.468 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:45:26.468 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:45:26.468 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:45:26.469 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:45:26.469 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:45:26.469 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:26.469 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:26.469 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:45:26.470 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:45:26.470 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:45:26.470 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:45:26.470 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:45:26.470 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:45:26.470 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:45:26.471 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:45:26.471 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:45:26.471 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:45:26.471 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:26.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:26.471 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:45:26.471 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:45:26.472 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:45:26.472 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:45:26.472 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:45:26.472 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:45:26.472 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:45:26.472 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:45:26.473 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:45:26.473 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:45:26.473 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:26.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:26.474 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:45:26.474 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:45:26.474 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:45:26.474 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:45:26.474 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:45:26.474 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:45:26.474 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:45:26.475 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:45:26.475 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:45:26.475 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:45:26.475 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:26.475 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:26.475 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:45:26.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:26.476 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:45:26.476 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:45:26.476 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:45:26.476 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:26.476 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:45:26.477 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:45:31.479 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:45:31.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:45:31.480 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:45:31.521 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:45:31.522 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:45:31.790 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:45:31.790 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:45:31.790 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:45:31.790 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:45:31.791 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:45:31.791 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:45:31.791 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:45:31.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:31.791 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:45:31.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:45:31.791 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:45:31.792 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:45:31.792 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:45:31.792 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:45:31.792 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:45:31.792 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:45:31.792 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:45:31.792 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:45:31.793 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:45:31.793 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:31.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:31.793 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:45:31.793 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:45:31.793 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:45:31.793 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:45:31.794 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:45:31.794 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:45:31.794 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:45:31.794 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:45:31.794 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:45:31.794 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:45:31.794 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:31.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:31.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:45:31.795 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:45:31.795 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:45:31.795 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:45:31.795 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:45:31.795 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:45:31.795 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:45:31.796 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:45:31.796 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:45:31.796 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:45:31.796 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:31.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:31.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:45:31.796 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:45:31.796 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:45:31.797 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:45:31.797 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:45:31.797 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:45:31.797 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:45:31.797 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:45:31.797 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:45:31.797 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:45:31.798 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:31.798 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:31.798 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:45:31.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:31.798 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:45:31.798 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:45:31.798 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:45:31.799 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:31.799 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:45:31.800 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:45:36.801 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:45:36.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:45:36.803 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:45:36.843 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:45:36.843 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:45:37.208 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:45:37.209 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:45:37.209 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:45:37.209 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:45:37.209 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:45:37.210 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:45:37.210 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:45:37.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:37.210 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:45:37.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:45:37.210 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:45:37.211 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:45:37.211 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:45:37.211 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:45:37.211 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:45:37.211 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:45:37.211 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:45:37.212 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:45:37.212 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:45:37.212 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:37.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:37.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:45:37.212 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:45:37.213 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:45:37.213 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:45:37.213 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:45:37.213 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:45:37.213 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:45:37.214 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:45:37.214 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:45:37.214 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:45:37.214 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:37.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:37.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:45:37.215 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:45:37.215 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:45:37.215 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:45:37.215 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:45:37.216 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:45:37.216 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:45:37.216 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:45:37.216 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:45:37.216 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:45:37.217 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:37.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:37.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:45:37.217 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:45:37.217 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:45:37.217 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:45:37.218 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:45:37.218 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:45:37.218 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:45:37.218 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:45:37.218 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:45:37.218 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:45:37.219 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:37.219 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:37.219 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:45:37.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:37.219 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:45:37.219 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:45:37.220 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:45:37.220 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:37.220 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:45:37.221 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:45:42.222 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:45:42.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:45:42.224 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:45:42.267 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:45:42.267 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:45:42.634 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:45:42.635 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:45:42.635 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:45:42.635 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:45:42.635 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:45:42.635 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:45:42.635 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:45:42.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:42.636 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:45:42.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:45:42.636 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:45:42.636 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:45:42.636 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:45:42.636 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:45:42.637 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:45:42.637 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:45:42.637 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:45:42.637 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:45:42.637 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:45:42.637 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:42.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:42.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:45:42.638 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:45:42.638 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:45:42.638 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:45:42.638 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:45:42.638 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:45:42.638 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:45:42.639 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:45:42.639 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:45:42.639 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:45:42.639 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:42.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:42.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:45:42.639 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:45:42.639 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:45:42.640 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:45:42.640 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:45:42.640 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:45:42.640 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:45:42.640 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:45:42.640 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:45:42.640 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:45:42.641 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:42.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:42.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:45:42.641 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:45:42.641 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:45:42.641 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:45:42.641 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:45:42.641 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:45:42.642 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:45:42.642 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:45:42.642 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:45:42.642 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:45:42.642 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:42.642 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:42.642 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:45:42.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:42.643 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:45:42.643 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:45:42.643 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:45:42.643 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:42.643 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:45:42.644 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:45:47.645 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:45:47.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:45:47.647 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:45:47.685 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:45:47.685 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:45:48.051 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:45:48.052 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:45:48.052 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:45:48.052 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:45:48.053 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:45:48.053 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:45:48.054 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:45:48.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:48.054 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:45:48.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:45:48.055 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:45:48.055 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:45:48.055 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:45:48.056 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:45:48.056 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:45:48.056 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:45:48.057 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:45:48.057 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:45:48.057 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:45:48.057 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:48.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:48.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:45:48.058 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:45:48.059 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:45:48.059 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:45:48.059 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:45:48.060 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:45:48.060 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:45:48.060 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:45:48.060 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:45:48.061 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:45:48.061 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:48.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:48.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:45:48.062 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:45:48.062 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:45:48.062 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:45:48.063 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:45:48.063 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:45:48.063 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:45:48.063 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:45:48.064 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:45:48.064 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:45:48.064 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:48.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:48.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:45:48.065 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:45:48.066 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:45:48.066 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:45:48.066 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:45:48.066 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:45:48.067 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:45:48.067 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:45:48.067 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:45:48.068 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:45:48.068 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:48.068 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:48.068 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:45:48.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:48.069 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:45:48.069 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:45:48.070 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:45:48.070 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:48.070 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:45:48.072 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:45:53.073 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:45:53.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:45:53.074 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:45:53.115 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:45:53.115 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:45:53.482 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:45:53.482 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:45:53.482 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:45:53.483 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:45:53.483 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:45:53.483 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:45:53.483 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:45:53.483 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:53.483 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:45:53.484 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:45:53.484 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:45:53.484 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:45:53.484 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:45:53.484 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:45:53.484 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:45:53.484 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:45:53.485 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:45:53.485 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:45:53.485 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:45:53.485 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:53.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:53.485 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:45:53.485 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:45:53.486 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:45:53.486 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:45:53.486 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:45:53.486 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:45:53.486 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:45:53.486 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:45:53.487 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:45:53.487 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:45:53.487 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:53.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:53.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:45:53.487 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:45:53.487 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:45:53.488 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:45:53.488 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:45:53.488 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:45:53.488 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:45:53.488 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:45:53.488 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:45:53.488 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:45:53.489 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:53.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:53.489 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:45:53.489 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:45:53.489 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:45:53.490 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:45:53.490 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:45:53.490 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:45:53.490 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:45:53.490 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:45:53.491 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:45:53.491 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:45:53.491 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:53.491 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:53.491 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:45:53.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:53.491 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:45:53.492 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:45:53.492 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:45:53.492 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:53.492 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:45:53.493 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:45:58.495 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:45:58.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:45:58.758 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:45:58.798 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:45:58.798 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:45:59.264 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:45:59.264 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:45:59.264 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:45:59.265 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:45:59.265 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:45:59.265 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:45:59.265 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:45:59.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:59.266 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:45:59.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:45:59.266 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:45:59.266 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:45:59.266 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:45:59.266 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:45:59.267 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:45:59.267 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:45:59.267 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:45:59.267 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:45:59.267 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:45:59.267 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:59.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:59.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:45:59.268 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:45:59.268 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:45:59.268 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:45:59.268 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:45:59.269 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:45:59.269 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:45:59.269 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:45:59.269 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:45:59.269 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:45:59.270 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:59.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:59.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:45:59.270 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:45:59.270 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:45:59.271 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:45:59.271 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:45:59.271 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:45:59.271 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:45:59.271 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:45:59.271 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:45:59.272 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:45:59.272 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:59.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:59.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:45:59.272 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:45:59.272 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:45:59.273 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:45:59.273 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:45:59.273 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:45:59.273 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:45:59.273 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:45:59.273 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:45:59.274 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:45:59.274 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:59.274 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:59.274 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:45:59.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:45:59.274 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:45:59.275 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:45:59.275 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:45:59.275 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:45:59.275 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:45:59.276 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:46:04.278 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:46:04.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:46:04.279 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:46:04.319 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:46:04.319 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:46:04.692 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:46:04.692 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:46:04.693 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:46:04.693 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:46:04.693 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:46:04.693 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:46:04.693 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:46:04.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:04.694 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:46:04.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:46:04.694 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:46:04.695 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:46:04.695 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:46:04.695 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:46:04.695 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:46:04.696 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:46:04.696 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:46:04.696 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:46:04.696 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:46:04.697 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:04.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:04.697 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:46:04.697 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:46:04.698 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:46:04.698 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:46:04.699 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:46:04.699 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:46:04.699 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:46:04.699 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:46:04.699 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:46:04.700 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:46:04.700 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:04.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:04.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:46:04.701 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:46:04.701 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:46:04.701 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:46:04.701 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:46:04.701 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:46:04.702 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:46:04.702 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:46:04.702 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:46:04.702 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:46:04.703 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:04.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:04.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:46:04.703 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:46:04.703 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:46:04.703 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:46:04.704 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:46:04.704 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:46:04.704 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:46:04.704 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:46:04.704 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:46:04.704 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:46:04.705 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:04.705 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:04.705 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:46:04.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:04.705 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:46:04.706 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:46:04.706 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:46:04.706 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:04.706 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:46:04.708 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:46:09.709 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:46:09.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:46:09.711 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:46:09.751 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:46:09.752 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:46:10.113 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:46:10.113 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:46:10.113 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:46:10.113 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:46:10.114 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:46:10.114 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:46:10.114 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:46:10.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:10.114 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:46:10.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:46:10.115 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:46:10.115 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:46:10.115 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:46:10.115 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:46:10.115 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:46:10.116 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:46:10.116 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:46:10.116 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:46:10.116 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:46:10.116 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:10.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:10.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:46:10.117 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:46:10.117 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:46:10.117 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:46:10.117 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:46:10.117 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:46:10.117 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:46:10.117 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:46:10.118 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:46:10.118 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:46:10.118 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:10.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:10.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:46:10.119 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:46:10.119 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:46:10.119 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:46:10.119 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:46:10.119 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:46:10.119 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:46:10.119 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:46:10.120 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:46:10.120 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:46:10.120 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:10.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:10.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:46:10.121 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:46:10.121 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:46:10.121 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:46:10.121 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:46:10.121 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:46:10.122 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:46:10.122 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:46:10.122 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:46:10.122 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:46:10.122 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:10.122 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:10.123 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:46:10.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:10.123 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:46:10.123 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:46:10.123 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:46:10.123 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:10.123 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:46:10.125 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:46:15.125 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:46:15.126 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:46:15.128 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:46:15.171 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:46:15.172 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:46:15.543 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:46:15.544 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:46:15.544 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:46:15.544 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:46:15.545 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:46:15.545 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:46:15.545 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:46:15.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:15.546 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:46:15.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:46:15.546 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:46:15.547 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:46:15.547 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:46:15.547 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:46:15.547 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:46:15.548 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:46:15.548 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:46:15.548 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:46:15.549 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:46:15.549 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:15.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:15.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:46:15.550 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:46:15.550 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:46:15.550 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:46:15.551 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:46:15.551 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:46:15.551 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:46:15.551 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:46:15.552 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:46:15.552 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:46:15.552 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:15.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:15.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:46:15.553 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:46:15.553 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:46:15.554 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:46:15.554 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:46:15.554 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:46:15.554 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:46:15.555 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:46:15.555 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:46:15.555 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:46:15.556 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:15.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:15.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:46:15.556 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:46:15.556 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:46:15.557 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:46:15.557 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:46:15.557 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:46:15.557 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:46:15.557 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:46:15.558 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:46:15.558 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:46:15.558 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:15.559 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:15.559 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:46:15.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:15.560 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:46:15.560 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:46:15.560 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:46:15.560 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:15.561 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:46:15.562 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:46:20.564 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:46:20.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:46:20.565 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:46:20.603 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:46:20.604 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:46:20.972 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:46:20.972 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:46:20.972 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:46:20.973 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:46:20.973 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:46:20.973 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:46:20.973 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:46:20.973 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:20.973 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:46:20.973 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:46:20.974 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:46:20.974 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:46:20.974 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:46:20.974 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:46:20.974 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:46:20.974 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:46:20.974 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:46:20.975 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:46:20.975 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:46:20.975 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:20.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:20.975 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:46:20.975 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:46:20.975 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:46:20.975 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:46:20.976 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:46:20.976 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:46:20.976 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:46:20.976 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:46:20.976 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:46:20.976 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:46:20.976 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:20.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:20.977 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:46:20.977 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:46:20.977 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:46:20.977 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:46:20.977 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:46:20.977 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:46:20.977 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:46:20.978 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:46:20.978 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:46:20.978 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:46:20.978 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:20.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:20.978 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:46:20.978 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:46:20.979 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:46:20.979 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:46:20.979 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:46:20.979 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:46:20.979 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:46:20.979 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:46:20.979 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:46:20.980 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:46:20.980 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:20.980 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:20.980 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:46:20.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:20.981 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:46:20.981 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:46:20.981 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:46:20.981 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:20.981 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:46:20.982 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:46:25.983 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:46:25.984 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:46:25.984 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:46:26.026 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:46:26.026 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:46:26.398 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:46:26.398 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:46:26.399 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:46:26.399 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:46:26.399 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:46:26.399 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:46:26.399 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:46:26.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:26.399 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:46:26.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:46:26.400 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:46:26.400 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:46:26.400 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:46:26.400 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:46:26.400 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:46:26.401 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:46:26.401 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:46:26.401 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:46:26.401 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:46:26.401 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:26.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:26.401 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:46:26.402 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:46:26.402 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:46:26.402 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:46:26.402 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:46:26.402 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:46:26.402 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:46:26.402 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:46:26.403 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:46:26.403 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:46:26.403 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:26.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:26.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:46:26.403 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:46:26.403 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:46:26.404 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:46:26.404 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:46:26.404 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:46:26.404 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:46:26.404 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:46:26.404 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:46:26.404 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:46:26.405 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:26.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:26.405 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:46:26.405 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:46:26.405 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:46:26.405 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:46:26.405 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:46:26.406 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:46:26.406 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:46:26.406 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:46:26.406 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:46:26.406 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:46:26.407 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:26.407 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:26.407 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:46:26.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:26.408 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:46:26.408 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:46:26.408 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:46:26.408 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:26.408 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:46:26.409 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:46:31.411 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:46:31.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:46:31.413 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:46:31.454 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:46:31.455 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:46:31.825 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:46:31.826 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:46:31.826 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:46:31.826 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:46:31.827 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:46:31.827 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:46:31.827 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:46:31.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:31.827 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:46:31.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:46:31.829 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:46:31.829 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:46:31.829 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:46:31.829 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:46:31.829 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:46:31.830 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:46:31.830 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:46:31.831 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:46:31.831 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:46:31.831 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:31.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:31.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:46:31.832 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:46:31.832 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:46:31.832 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:46:31.832 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:46:31.832 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:46:31.833 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:46:31.833 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:46:31.833 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:46:31.833 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:46:31.833 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:31.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:31.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:46:31.834 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:46:31.834 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:46:31.834 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:46:31.835 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:46:31.835 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:46:31.835 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:46:31.835 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:46:31.835 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:46:31.836 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:46:31.836 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:31.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:31.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:46:31.836 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:46:31.837 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:46:31.837 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:46:31.837 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:46:31.837 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:46:31.837 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:46:31.838 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:46:31.838 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:46:31.838 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:46:31.838 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:31.839 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:31.839 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:46:31.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:31.840 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:46:31.840 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:46:31.840 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:46:31.840 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:31.841 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:46:31.842 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:46:36.844 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:46:36.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:46:36.846 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:46:36.890 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:46:36.890 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:46:37.259 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:46:37.260 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:46:37.261 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:46:37.261 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:46:37.264 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:46:37.264 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:46:37.266 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:46:37.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:37.267 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:46:37.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:46:37.268 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:46:37.268 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:46:37.269 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:46:37.269 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:46:37.272 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:46:37.273 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:46:37.274 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:46:37.275 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:46:37.276 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:46:37.276 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:37.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:37.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:46:37.277 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:46:37.277 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:46:37.277 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:46:37.277 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:46:37.278 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:46:37.278 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:46:37.278 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:46:37.278 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:46:37.278 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:46:37.278 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:37.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:37.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:46:37.279 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:46:37.279 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:46:37.279 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:46:37.279 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:46:37.279 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:46:37.279 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:46:37.280 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:46:37.280 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:46:37.280 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:46:37.280 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:37.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:37.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:46:37.280 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:46:37.280 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:46:37.281 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:46:37.281 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:46:37.281 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:46:37.281 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:46:37.281 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:46:37.281 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:46:37.281 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:46:37.282 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:37.282 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:37.282 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:46:37.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:37.282 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:46:37.282 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:46:37.282 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:46:37.283 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:37.283 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:46:37.284 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:46:42.285 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:46:42.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:46:42.287 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:46:42.327 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:46:42.327 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:46:42.690 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:46:42.690 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:46:42.691 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:46:42.691 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:46:42.692 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:46:42.693 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:46:42.693 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:46:42.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:42.693 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:46:42.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:46:42.693 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:46:42.694 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:46:42.694 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:46:42.694 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:46:42.694 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:46:42.694 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:46:42.694 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:46:42.695 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:46:42.695 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:46:42.695 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:42.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:42.695 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:46:42.695 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:46:42.695 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:46:42.696 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:46:42.696 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:46:42.696 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:46:42.696 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:46:42.696 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:46:42.696 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:46:42.696 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:46:42.697 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:42.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:42.697 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:46:42.697 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:46:42.697 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:46:42.697 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:46:42.697 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:46:42.698 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:46:42.698 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:46:42.698 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:46:42.698 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:46:42.699 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:46:42.699 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:42.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:42.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:46:42.699 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:46:42.699 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:46:42.700 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:46:42.700 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:46:42.700 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:46:42.700 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:46:42.700 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:46:42.700 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:46:42.700 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:46:42.701 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:42.701 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:42.701 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:46:42.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:42.701 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:46:42.701 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:46:42.701 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:46:42.702 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:42.702 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:46:42.703 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:46:47.704 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:46:47.705 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:46:47.705 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:46:47.746 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:46:47.747 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:46:48.015 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:46:48.016 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:46:48.016 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:46:48.016 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:46:48.016 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:46:48.016 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:46:48.017 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:46:48.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:48.017 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:46:48.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:46:48.017 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:46:48.017 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:46:48.018 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:46:48.018 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:46:48.018 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:46:48.018 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:46:48.018 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:46:48.018 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:46:48.019 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:46:48.019 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:48.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:48.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:46:48.019 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:46:48.019 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:46:48.020 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:46:48.020 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:46:48.020 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:46:48.020 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:46:48.020 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:46:48.021 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:46:48.021 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:46:48.021 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:48.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:48.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:46:48.022 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:46:48.022 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:46:48.022 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:46:48.022 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:46:48.022 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:46:48.022 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:46:48.022 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:46:48.023 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:46:48.023 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:46:48.023 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:48.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:48.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:46:48.023 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:46:48.024 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:46:48.024 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:46:48.024 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:46:48.024 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:46:48.024 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:46:48.024 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:46:48.025 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:46:48.025 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:46:48.025 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:48.025 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:48.025 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:46:48.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:48.026 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:46:48.026 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:46:48.026 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:46:48.026 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:48.026 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:46:48.027 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:46:53.029 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:46:53.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:46:53.030 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:46:53.070 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:46:53.071 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:46:53.439 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:46:53.440 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:46:53.440 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:46:53.440 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:46:53.440 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:46:53.441 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:46:53.441 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:46:53.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:53.441 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:46:53.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:46:53.441 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:46:53.441 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:46:53.442 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:46:53.442 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:46:53.442 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:46:53.442 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:46:53.443 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:46:53.443 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:46:53.443 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:46:53.443 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:53.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:53.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:46:53.443 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:46:53.444 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:46:53.444 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:46:53.444 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:46:53.444 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:46:53.444 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:46:53.444 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:46:53.444 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:46:53.445 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:46:53.445 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:53.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:53.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:46:53.446 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:46:53.446 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:46:53.446 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:46:53.446 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:46:53.447 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:46:53.447 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:46:53.447 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:46:53.447 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:46:53.447 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:46:53.447 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:53.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:53.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:46:53.448 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:46:53.448 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:46:53.448 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:46:53.448 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:46:53.448 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:46:53.449 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:46:53.449 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:46:53.449 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:46:53.449 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:46:53.450 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:53.450 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:53.450 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:46:53.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:53.450 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:46:53.450 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:46:53.451 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:46:53.451 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:53.451 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:46:53.452 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:46:58.454 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:46:58.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:46:58.455 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:46:58.495 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:46:58.495 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:46:58.861 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:46:58.862 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:46:58.862 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:46:58.862 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:46:58.862 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:46:58.863 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:46:58.863 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:46:58.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:58.863 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:46:58.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:46:58.863 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:46:58.863 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:46:58.864 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:46:58.864 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:46:58.864 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:46:58.864 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:46:58.864 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:46:58.864 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:46:58.864 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:46:58.865 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:58.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:58.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:46:58.865 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:46:58.865 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:46:58.865 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:46:58.865 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:46:58.865 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:46:58.866 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:46:58.866 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:46:58.866 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:46:58.866 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:46:58.866 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:58.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:58.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:46:58.867 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:46:58.867 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:46:58.867 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:46:58.867 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:46:58.867 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:46:58.867 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:46:58.867 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:46:58.868 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:46:58.868 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:46:58.868 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:58.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:58.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:46:58.868 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:46:58.868 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:46:58.868 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:46:58.869 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:46:58.869 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:46:58.869 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:46:58.869 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:46:58.869 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:46:58.869 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:46:58.869 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:58.870 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:58.870 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:46:58.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:46:58.870 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:46:58.870 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:46:58.870 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:46:58.870 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:46:58.871 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:46:58.871 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:47:03.873 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:47:03.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:47:03.875 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:47:03.917 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:47:03.917 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:47:04.285 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:47:04.286 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:47:04.286 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:47:04.286 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:47:04.286 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:47:04.287 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:47:04.287 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:47:04.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:04.287 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:47:04.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:04.288 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:47:04.288 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:04.288 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:04.288 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:47:04.288 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:04.288 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:47:04.289 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:04.289 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:47:04.289 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:47:04.289 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:04.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:04.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:04.290 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:47:04.290 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:04.290 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:04.290 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:47:04.290 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:04.290 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:47:04.291 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:04.291 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:47:04.291 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:47:04.291 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:04.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:04.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:04.292 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:47:04.292 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:04.292 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:04.292 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:47:04.292 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:04.292 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:47:04.293 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:04.293 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:47:04.293 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:47:04.293 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:04.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:04.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:04.294 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:47:04.294 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:04.294 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:04.294 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:47:04.295 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:04.295 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:47:04.295 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:04.295 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:47:04.295 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:47:04.295 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:04.295 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:04.296 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:47:04.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:04.296 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:47:04.296 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:47:04.296 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:47:04.296 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:04.297 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:47:04.298 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:47:09.299 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:47:09.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:47:09.300 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:47:09.341 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:47:09.342 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:47:09.706 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:47:09.708 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:47:09.708 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:47:09.708 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:47:09.708 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:47:09.708 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:47:09.708 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:47:09.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:09.709 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:47:09.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:09.709 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:47:09.709 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:09.709 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:09.709 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:47:09.709 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:09.710 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:47:09.710 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:09.710 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:47:09.710 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:47:09.710 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:09.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:09.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:09.711 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:47:09.711 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:09.711 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:09.711 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:47:09.711 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:09.711 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:47:09.711 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:09.712 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:47:09.712 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:47:09.712 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:09.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:09.712 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:09.712 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:47:09.712 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:09.712 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:09.713 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:47:09.713 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:09.713 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:47:09.713 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:09.713 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:47:09.713 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:47:09.713 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:09.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:09.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:09.714 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:47:09.714 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:09.714 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:09.714 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:47:09.714 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:09.715 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:47:09.715 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:09.715 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:47:09.715 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:47:09.715 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:09.715 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:09.715 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:47:09.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:09.716 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:47:09.716 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:47:09.716 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:47:09.716 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:09.716 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:47:09.717 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:47:14.719 INFO:tasks.rgw_multisite_tests:test s3 topics with secret set/get/delete on master ... FAIL 2020-11-09T18:47:14.720 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:47:14.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:47:14.724 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:47:14.765 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:47:14.766 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:47:15.019 INFO:teuthology.orchestra.run.smithi051.stderr:No --period given, using current period=f6258aba-300a-4d5d-ad93-a7c198a0e0a1 2020-11-09T18:47:15.054 INFO:teuthology.orchestra.run.smithi051.stdout:[ 2020-11-09T18:47:15.054 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:47:15.055 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604946340.774139_451.1", 2020-11-09T18:47:15.055 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "2020-11-09T18:25:40.774139Z" 2020-11-09T18:47:15.055 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:47:15.055 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:47:15.055 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:47:15.055 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:47:15.056 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:47:15.056 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:47:15.056 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:47:15.056 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:47:15.056 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:47:15.056 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:47:15.058 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:47:15.058 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:47:15.058 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:47:15.059 INFO:teuthology.orchestra.run.smithi051.stdout:] 2020-11-09T18:47:15.060 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-09T18:47:15.061 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:47:15.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:47:15.062 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:47:15.102 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:47:15.102 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:47:15.382 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:47:15.382 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:47:15.383 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:47:15.383 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:47:15.384 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:47:15.385 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:47:15.385 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:47:15.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:15.385 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:47:15.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:15.385 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:47:15.386 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:15.386 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:15.386 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946340.774139_451.1", 2020-11-09T18:47:15.386 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:15.386 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:47:15.387 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:15.387 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:40.774139Z", 2020-11-09T18:47:15.387 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:47:15.387 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:15.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:15.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:15.387 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:47:15.388 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:15.388 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:15.388 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:47:15.388 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:15.388 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:47:15.388 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:15.388 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:47:15.389 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:47:15.389 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:15.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:15.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:15.389 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:47:15.389 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:15.389 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:15.389 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:47:15.390 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:15.390 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:47:15.390 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:15.390 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:47:15.390 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:47:15.390 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:15.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:15.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:15.391 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:47:15.391 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:15.391 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:15.391 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:47:15.391 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:15.391 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:47:15.392 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:15.392 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:47:15.392 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:47:15.392 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:15.392 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:15.392 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:47:15.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:15.392 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:47:15.393 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:47:15.393 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:47:15.393 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:15.393 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:47:15.394 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-09T18:47:15.395 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-09T18:47:15.395 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:47:15.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:47:15.396 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:47:15.437 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:47:15.437 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:47:15.691 INFO:teuthology.orchestra.run.smithi051.stdout:{ 2020-11-09T18:47:15.691 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_status": { 2020-11-09T18:47:15.691 INFO:teuthology.orchestra.run.smithi051.stdout: "info": { 2020-11-09T18:47:15.692 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "sync", 2020-11-09T18:47:15.692 INFO:teuthology.orchestra.run.smithi051.stdout: "num_shards": 4, 2020-11-09T18:47:15.692 INFO:teuthology.orchestra.run.smithi051.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:47:15.692 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 4 2020-11-09T18:47:15.692 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:47:15.692 INFO:teuthology.orchestra.run.smithi051.stdout: "markers": [ 2020-11-09T18:47:15.692 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:47:15.693 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 0, 2020-11-09T18:47:15.693 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:47:15.693 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:47:15.693 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604946340.774139_451.1", 2020-11-09T18:47:15.693 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:47:15.693 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 1, 2020-11-09T18:47:15.693 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:47:15.694 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:25:40.774139Z", 2020-11-09T18:47:15.694 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 4 2020-11-09T18:47:15.694 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:47:15.694 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:47:15.694 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:47:15.694 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 1, 2020-11-09T18:47:15.694 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:47:15.695 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:47:15.695 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:47:15.695 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:47:15.695 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:47:15.695 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:47:15.695 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:47:15.695 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:47:15.696 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:47:15.696 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:47:15.696 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:47:15.696 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 2, 2020-11-09T18:47:15.696 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:47:15.696 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:47:15.696 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:47:15.696 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:47:15.697 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:47:15.697 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:47:15.697 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:47:15.697 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:47:15.697 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:47:15.697 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:47:15.697 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:47:15.698 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 3, 2020-11-09T18:47:15.698 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:47:15.698 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:47:15.698 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:47:15.698 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:47:15.698 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:47:15.698 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:47:15.699 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:47:15.699 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:47:15.699 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:47:15.699 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:47:15.699 INFO:teuthology.orchestra.run.smithi051.stdout: ] 2020-11-09T18:47:15.699 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:47:15.699 INFO:teuthology.orchestra.run.smithi051.stdout: "full_sync": { 2020-11-09T18:47:15.699 INFO:teuthology.orchestra.run.smithi051.stdout: "total": 1, 2020-11-09T18:47:15.700 INFO:teuthology.orchestra.run.smithi051.stdout: "complete": 1 2020-11-09T18:47:15.700 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:47:15.700 INFO:teuthology.orchestra.run.smithi051.stdout:} 2020-11-09T18:47:15.701 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-09T18:47:15.701 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-09T18:47:15.702 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:47:15.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:47:15.703 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:47:15.743 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:47:15.744 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:47:16.014 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:47:16.015 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:47:16.015 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:47:16.015 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:47:16.015 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:47:16.016 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:47:16.016 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:47:16.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:16.016 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:47:16.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:16.016 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:47:16.017 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:16.017 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:16.017 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:47:16.017 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:16.017 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:47:16.017 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:16.018 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:47:16.018 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:47:16.018 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:16.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:16.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:16.018 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:47:16.018 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:16.019 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:16.019 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:47:16.019 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:16.019 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:47:16.019 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:16.019 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:47:16.019 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:47:16.019 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:16.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:16.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:16.020 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:47:16.020 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:16.020 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:16.020 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:47:16.020 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:16.021 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:47:16.021 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:16.021 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:47:16.021 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:47:16.021 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:16.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:16.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:16.022 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:47:16.022 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:16.022 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:16.023 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:47:16.023 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:16.023 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:47:16.023 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:16.023 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:47:16.023 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:47:16.023 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:16.024 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:16.024 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:47:16.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:16.024 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:47:16.024 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:47:16.024 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:47:16.024 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:16.025 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:47:16.026 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:47:21.027 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:47:21.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:47:21.029 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:47:21.069 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:47:21.070 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:47:21.435 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:47:21.436 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:47:21.436 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:47:21.436 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:47:21.436 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:47:21.436 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:47:21.436 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:47:21.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:21.437 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:47:21.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:21.437 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:47:21.437 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:21.437 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:21.437 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:47:21.438 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:21.438 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:47:21.438 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:21.438 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:47:21.438 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:47:21.438 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:21.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:21.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:21.439 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:47:21.439 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:21.439 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:21.439 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:47:21.439 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:21.439 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:47:21.440 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:21.440 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:47:21.440 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:47:21.440 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:21.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:21.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:21.440 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:47:21.440 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:21.441 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:21.441 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:47:21.441 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:21.441 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:47:21.441 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:21.441 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:47:21.441 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:47:21.442 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:21.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:21.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:21.442 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:47:21.442 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:21.442 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:21.442 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:47:21.443 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:21.443 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:47:21.443 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:21.443 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:47:21.443 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:47:21.443 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:21.443 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:21.443 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:47:21.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:21.444 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:47:21.444 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:47:21.444 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:47:21.444 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:21.445 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:47:21.446 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:47:26.447 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:47:26.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:47:26.449 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:47:26.490 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:47:26.490 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:47:26.867 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:47:26.867 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:47:26.867 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:47:26.867 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:47:26.867 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:47:26.868 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:47:26.868 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:47:26.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:26.868 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:47:26.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:26.868 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:47:26.869 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:26.869 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:26.869 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:47:26.869 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:26.869 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:47:26.869 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:26.869 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:47:26.870 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:47:26.870 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:26.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:26.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:26.870 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:47:26.870 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:26.871 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:26.871 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:47:26.871 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:26.871 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:47:26.871 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:26.872 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:47:26.872 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:47:26.872 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:26.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:26.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:26.872 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:47:26.872 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:26.873 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:26.873 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:47:26.873 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:26.873 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:47:26.873 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:26.873 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:47:26.873 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:47:26.874 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:26.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:26.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:26.874 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:47:26.874 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:26.874 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:26.875 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:47:26.875 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:26.875 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:47:26.875 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:26.875 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:47:26.875 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:47:26.876 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:26.876 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:26.876 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:47:26.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:26.877 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:47:26.877 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:47:26.877 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:47:26.877 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:26.877 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:47:26.878 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:47:31.882 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:47:31.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:47:31.883 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:47:31.924 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:47:31.925 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:47:32.292 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:47:32.293 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:47:32.293 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:47:32.293 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:47:32.293 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:47:32.294 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:47:32.294 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:47:32.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:32.295 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:47:32.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:32.295 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:47:32.295 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:32.295 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:32.296 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:47:32.296 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:32.296 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:47:32.296 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:32.296 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:47:32.296 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:47:32.297 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:32.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:32.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:32.297 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:47:32.297 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:32.298 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:32.298 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:47:32.298 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:32.298 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:47:32.298 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:32.299 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:47:32.299 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:47:32.299 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:32.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:32.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:32.299 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:47:32.300 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:32.300 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:32.300 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:47:32.300 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:32.300 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:47:32.301 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:32.301 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:47:32.301 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:47:32.301 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:32.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:32.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:32.302 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:47:32.302 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:32.302 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:32.302 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:47:32.302 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:32.303 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:47:32.303 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:32.303 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:47:32.303 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:47:32.303 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:32.304 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:32.304 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:47:32.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:32.304 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:47:32.304 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:47:32.305 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:47:32.305 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:32.305 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:47:32.306 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:47:37.307 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:47:37.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:47:37.308 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:47:37.350 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:47:37.351 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:47:37.723 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:47:37.723 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:47:37.723 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:47:37.723 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:47:37.723 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:47:37.724 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:47:37.724 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:47:37.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:37.724 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:47:37.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:37.724 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:47:37.724 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:37.725 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:37.725 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:47:37.725 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:37.725 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:47:37.725 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:37.725 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:47:37.725 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:47:37.726 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:37.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:37.726 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:37.726 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:47:37.726 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:37.726 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:37.726 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:47:37.727 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:37.727 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:47:37.727 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:37.727 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:47:37.727 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:47:37.727 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:37.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:37.727 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:37.728 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:47:37.728 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:37.728 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:37.728 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:47:37.728 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:37.728 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:47:37.729 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:37.729 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:47:37.729 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:47:37.730 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:37.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:37.730 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:37.730 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:47:37.730 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:37.730 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:37.730 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:47:37.731 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:37.731 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:47:37.731 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:37.731 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:47:37.731 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:47:37.731 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:37.731 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:37.732 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:47:37.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:37.732 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:47:37.732 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:47:37.732 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:47:37.732 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:37.732 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:47:37.734 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:47:42.735 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:47:42.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:47:42.737 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:47:42.778 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:47:42.779 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:47:43.143 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:47:43.143 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:47:43.143 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:47:43.143 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:47:43.144 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:47:43.144 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:47:43.144 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:47:43.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:43.144 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:47:43.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:43.144 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:47:43.145 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:43.145 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:43.145 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:47:43.145 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:43.145 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:47:43.145 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:43.145 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:47:43.146 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:47:43.146 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:43.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:43.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:43.146 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:47:43.146 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:43.146 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:43.147 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:47:43.147 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:43.147 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:47:43.147 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:43.147 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:47:43.147 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:47:43.147 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:43.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:43.148 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:43.148 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:47:43.148 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:43.148 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:43.148 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:47:43.148 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:43.148 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:47:43.149 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:43.149 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:47:43.149 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:47:43.149 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:43.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:43.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:43.149 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:47:43.150 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:43.150 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:43.150 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:47:43.150 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:43.150 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:47:43.150 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:43.150 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:47:43.151 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:47:43.151 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:43.151 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:43.151 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:47:43.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:43.151 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:47:43.151 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:47:43.152 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:47:43.152 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:43.152 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:47:43.153 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:47:48.154 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:47:48.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:47:48.156 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:47:48.207 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:47:48.208 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:47:48.574 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:47:48.574 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:47:48.575 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:47:48.575 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:47:48.575 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:47:48.575 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:47:48.575 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:47:48.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:48.575 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:47:48.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:48.576 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:47:48.576 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:48.576 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:48.576 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:47:48.576 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:48.576 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:47:48.577 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:48.577 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:47:48.577 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:47:48.577 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:48.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:48.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:48.577 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:47:48.578 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:48.578 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:48.578 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:47:48.578 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:48.578 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:47:48.578 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:48.578 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:47:48.579 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:47:48.579 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:48.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:48.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:48.579 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:47:48.579 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:48.579 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:48.579 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:47:48.580 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:48.580 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:47:48.580 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:48.580 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:47:48.580 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:47:48.580 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:48.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:48.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:48.581 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:47:48.581 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:48.581 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:48.581 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:47:48.581 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:48.581 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:47:48.582 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:48.582 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:47:48.582 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:47:48.582 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:48.582 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:48.582 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:47:48.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:48.582 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:47:48.583 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:47:48.583 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:47:48.583 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:48.583 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:47:48.584 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:47:53.585 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:47:53.586 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:47:53.586 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:47:53.626 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:47:53.627 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:47:53.997 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:47:53.998 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:47:53.998 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:47:53.998 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:47:53.998 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:47:53.998 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:47:53.998 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:47:53.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:53.999 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:47:53.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:53.999 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:47:53.999 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:53.999 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:53.999 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:47:53.999 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:54.000 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:47:54.000 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:54.000 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:47:54.000 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:47:54.000 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:54.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:54.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:54.001 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:47:54.001 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:54.001 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:54.001 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:47:54.001 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:54.001 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:47:54.002 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:54.002 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:47:54.002 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:47:54.002 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:54.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:54.002 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:54.003 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:47:54.003 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:54.003 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:54.003 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:47:54.003 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:54.003 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:47:54.004 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:54.004 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:47:54.004 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:47:54.004 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:54.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:54.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:54.004 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:47:54.004 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:54.005 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:54.005 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:47:54.005 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:54.005 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:47:54.005 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:54.005 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:47:54.006 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:47:54.006 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:54.006 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:54.006 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:47:54.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:54.006 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:47:54.007 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:47:54.007 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:47:54.007 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:54.007 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:47:54.008 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:47:59.009 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:47:59.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:47:59.011 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:47:59.050 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:47:59.050 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:47:59.421 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:47:59.422 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:47:59.422 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:47:59.422 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:47:59.422 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:47:59.422 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:47:59.422 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:47:59.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:59.423 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:47:59.423 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:59.423 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:47:59.423 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:59.423 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:59.423 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:47:59.424 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:59.424 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:47:59.424 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:59.424 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:47:59.424 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:47:59.424 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:59.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:59.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:59.425 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:47:59.425 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:59.425 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:59.425 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:47:59.425 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:59.425 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:47:59.425 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:59.426 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:47:59.426 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:47:59.426 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:59.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:59.426 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:59.426 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:47:59.426 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:59.427 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:59.427 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:47:59.427 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:59.427 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:47:59.427 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:59.427 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:47:59.427 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:47:59.428 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:59.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:59.428 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:47:59.428 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:47:59.428 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:47:59.428 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:47:59.428 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:47:59.429 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:47:59.429 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:47:59.429 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:47:59.429 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:47:59.429 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:47:59.429 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:59.429 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:59.430 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:47:59.430 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:47:59.430 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:47:59.430 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:47:59.430 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:47:59.430 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:47:59.430 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:47:59.432 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:48:04.433 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:48:04.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:48:04.436 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:48:04.479 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:48:04.479 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:48:04.846 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:48:04.847 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:48:04.847 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:48:04.847 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:48:04.847 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:48:04.847 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:48:04.848 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:48:04.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:04.848 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:48:04.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:48:04.848 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:48:04.849 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:48:04.849 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:48:04.849 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:48:04.849 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:48:04.849 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:48:04.849 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:48:04.850 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:48:04.850 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:48:04.850 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:04.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:04.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:48:04.850 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:48:04.851 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:48:04.851 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:48:04.851 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:48:04.851 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:48:04.851 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:48:04.851 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:48:04.852 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:48:04.852 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:48:04.852 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:04.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:04.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:48:04.852 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:48:04.852 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:48:04.853 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:48:04.853 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:48:04.853 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:48:04.853 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:48:04.853 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:48:04.853 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:48:04.854 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:48:04.854 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:04.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:04.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:48:04.854 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:48:04.854 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:48:04.854 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:48:04.855 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:48:04.855 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:48:04.855 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:48:04.855 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:48:04.855 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:48:04.855 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:48:04.856 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:04.856 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:04.856 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:48:04.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:04.856 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:48:04.856 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:48:04.856 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:48:04.857 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:04.857 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:48:04.858 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:48:09.859 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:48:09.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:48:09.860 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:48:09.910 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:48:09.911 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:48:10.275 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:48:10.276 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:48:10.276 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:48:10.276 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:48:10.277 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:48:10.277 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:48:10.277 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:48:10.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:10.278 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:48:10.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:48:10.278 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:48:10.278 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:48:10.278 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:48:10.279 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:48:10.279 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:48:10.279 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:48:10.279 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:48:10.279 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:48:10.279 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:48:10.280 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:10.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:10.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:48:10.280 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:48:10.280 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:48:10.281 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:48:10.282 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:48:10.282 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:48:10.282 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:48:10.282 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:48:10.283 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:48:10.283 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:48:10.283 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:10.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:10.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:48:10.284 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:48:10.284 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:48:10.284 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:48:10.284 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:48:10.285 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:48:10.285 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:48:10.285 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:48:10.285 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:48:10.285 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:48:10.286 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:10.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:10.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:48:10.286 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:48:10.286 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:48:10.287 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:48:10.287 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:48:10.287 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:48:10.287 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:48:10.287 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:48:10.287 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:48:10.288 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:48:10.288 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:10.288 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:10.288 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:48:10.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:10.289 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:48:10.289 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:48:10.289 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:48:10.289 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:10.290 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:48:10.291 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:48:15.292 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:48:15.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:48:15.294 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:48:15.336 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:48:15.336 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:48:15.701 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:48:15.701 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:48:15.702 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:48:15.702 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:48:15.702 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:48:15.702 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:48:15.702 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:48:15.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:15.702 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:48:15.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:48:15.703 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:48:15.703 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:48:15.703 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:48:15.703 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:48:15.703 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:48:15.703 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:48:15.704 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:48:15.704 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:48:15.704 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:48:15.704 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:15.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:15.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:48:15.704 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:48:15.705 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:48:15.705 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:48:15.705 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:48:15.705 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:48:15.705 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:48:15.706 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:48:15.706 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:48:15.706 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:48:15.706 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:15.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:15.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:48:15.706 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:48:15.707 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:48:15.707 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:48:15.707 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:48:15.707 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:48:15.707 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:48:15.707 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:48:15.707 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:48:15.708 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:48:15.708 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:15.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:15.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:48:15.708 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:48:15.708 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:48:15.708 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:48:15.709 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:48:15.709 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:48:15.709 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:48:15.709 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:48:15.709 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:48:15.710 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:48:15.710 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:15.710 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:15.710 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:48:15.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:15.711 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:48:15.712 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:48:15.712 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:48:15.712 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:15.713 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:48:15.714 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:48:20.715 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:48:20.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:48:20.716 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:48:20.755 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:48:20.755 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:48:21.116 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:48:21.116 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:48:21.116 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:48:21.117 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:48:21.117 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:48:21.117 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:48:21.117 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:48:21.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:21.117 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:48:21.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:48:21.118 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:48:21.118 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:48:21.118 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:48:21.118 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:48:21.118 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:48:21.118 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:48:21.118 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:48:21.119 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:48:21.119 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:48:21.119 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:21.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:21.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:48:21.119 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:48:21.119 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:48:21.120 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:48:21.120 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:48:21.120 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:48:21.120 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:48:21.120 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:48:21.120 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:48:21.120 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:48:21.120 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:21.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:21.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:48:21.121 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:48:21.121 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:48:21.121 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:48:21.121 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:48:21.121 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:48:21.122 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:48:21.122 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:48:21.122 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:48:21.122 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:48:21.122 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:21.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:21.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:48:21.123 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:48:21.123 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:48:21.123 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:48:21.123 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:48:21.123 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:48:21.123 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:48:21.123 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:48:21.123 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:48:21.124 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:48:21.124 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:21.124 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:21.124 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:48:21.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:21.124 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:48:21.124 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:48:21.125 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:48:21.125 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:21.125 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:48:21.126 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:48:26.127 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:48:26.128 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:48:26.129 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:48:26.174 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:48:26.174 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:48:26.537 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:48:26.538 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:48:26.538 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:48:26.538 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:48:26.538 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:48:26.538 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:48:26.538 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:48:26.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:26.539 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:48:26.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:48:26.539 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:48:26.539 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:48:26.539 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:48:26.540 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:48:26.540 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:48:26.540 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:48:26.540 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:48:26.540 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:48:26.540 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:48:26.540 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:26.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:26.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:48:26.541 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:48:26.541 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:48:26.541 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:48:26.541 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:48:26.541 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:48:26.541 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:48:26.542 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:48:26.542 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:48:26.542 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:48:26.542 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:26.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:26.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:48:26.542 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:48:26.543 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:48:26.543 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:48:26.543 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:48:26.543 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:48:26.543 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:48:26.543 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:48:26.543 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:48:26.544 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:48:26.544 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:26.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:26.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:48:26.544 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:48:26.544 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:48:26.544 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:48:26.545 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:48:26.545 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:48:26.545 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:48:26.545 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:48:26.545 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:48:26.545 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:48:26.545 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:26.545 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:26.546 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:48:26.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:26.546 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:48:26.546 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:48:26.546 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:48:26.546 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:26.546 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:48:26.548 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:48:31.549 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:48:31.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:48:31.551 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:48:31.597 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:48:31.597 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:48:31.944 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:48:31.944 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:48:31.944 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:48:31.944 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:48:31.944 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:48:31.945 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:48:31.945 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:48:31.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:31.945 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:48:31.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:48:31.945 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:48:31.946 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:48:31.946 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:48:31.946 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:48:31.946 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:48:31.946 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:48:31.946 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:48:31.947 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:48:31.947 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:48:31.947 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:31.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:31.947 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:48:31.947 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:48:31.947 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:48:31.948 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:48:31.948 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:48:31.948 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:48:31.948 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:48:31.948 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:48:31.948 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:48:31.948 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:48:31.949 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:31.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:31.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:48:31.949 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:48:31.949 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:48:31.949 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:48:31.950 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:48:31.950 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:48:31.950 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:48:31.950 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:48:31.950 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:48:31.950 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:48:31.950 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:31.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:31.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:48:31.951 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:48:31.951 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:48:31.951 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:48:31.951 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:48:31.951 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:48:31.952 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:48:31.952 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:48:31.952 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:48:31.952 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:48:31.952 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:31.952 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:31.952 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:48:31.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:31.953 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:48:31.953 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:48:31.953 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:48:31.953 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:31.954 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:48:31.955 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:48:36.956 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:48:36.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:48:36.958 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:48:36.998 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:48:36.999 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:48:37.364 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:48:37.364 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:48:37.365 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:48:37.365 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:48:37.365 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:48:37.365 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:48:37.365 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:48:37.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:37.365 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:48:37.366 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:48:37.366 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:48:37.366 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:48:37.366 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:48:37.366 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:48:37.366 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:48:37.366 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:48:37.367 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:48:37.367 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:48:37.367 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:48:37.367 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:37.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:37.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:48:37.367 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:48:37.368 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:48:37.368 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:48:37.368 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:48:37.368 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:48:37.368 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:48:37.368 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:48:37.368 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:48:37.369 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:48:37.369 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:37.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:37.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:48:37.369 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:48:37.369 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:48:37.370 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:48:37.370 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:48:37.370 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:48:37.370 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:48:37.370 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:48:37.370 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:48:37.370 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:48:37.371 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:37.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:37.371 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:48:37.371 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:48:37.371 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:48:37.371 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:48:37.371 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:48:37.372 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:48:37.372 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:48:37.372 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:48:37.372 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:48:37.372 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:48:37.372 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:37.373 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:37.373 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:48:37.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:37.373 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:48:37.373 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:48:37.373 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:48:37.373 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:37.374 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:48:37.375 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:48:42.376 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:48:42.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:48:42.377 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:48:42.418 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:48:42.418 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:48:42.786 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:48:43.025 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:48:43.026 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:48:43.026 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:48:43.026 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:48:43.026 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:48:43.027 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:48:43.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:43.027 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:48:43.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:48:43.027 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:48:43.028 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:48:43.028 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:48:43.028 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:48:43.028 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:48:43.029 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:48:43.029 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:48:43.029 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:48:43.029 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:48:43.029 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:43.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:43.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:48:43.030 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:48:43.030 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:48:43.031 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:48:43.031 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:48:43.031 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:48:43.031 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:48:43.031 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:48:43.032 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:48:43.032 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:48:43.032 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:43.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:43.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:48:43.033 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:48:43.033 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:48:43.033 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:48:43.034 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:48:43.035 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:48:43.035 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:48:43.035 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:48:43.035 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:48:43.036 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:48:43.036 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:43.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:43.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:48:43.037 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:48:43.037 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:48:43.037 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:48:43.037 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:48:43.038 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:48:43.038 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:48:43.038 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:48:43.038 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:48:43.039 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:48:43.039 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:43.039 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:43.039 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:48:43.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:43.039 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:48:43.040 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:48:43.040 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:48:43.040 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:43.045 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:48:43.047 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:48:48.054 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:48:48.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:48:48.055 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:48:48.098 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:48:48.098 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:48:48.459 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:48:48.459 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:48:48.459 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:48:48.460 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:48:48.460 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:48:48.460 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:48:48.460 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:48:48.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:48.461 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:48:48.461 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:48:48.461 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:48:48.462 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:48:48.462 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:48:48.462 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:48:48.462 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:48:48.463 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:48:48.463 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:48:48.463 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:48:48.463 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:48:48.463 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:48.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:48.464 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:48:48.464 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:48:48.464 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:48:48.464 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:48:48.465 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:48:48.465 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:48:48.465 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:48:48.465 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:48:48.466 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:48:48.466 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:48:48.466 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:48.466 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:48.467 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:48:48.467 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:48:48.467 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:48:48.467 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:48:48.468 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:48:48.468 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:48:48.468 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:48:48.468 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:48:48.469 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:48:48.469 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:48:48.469 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:48.469 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:48.470 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:48:48.470 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:48:48.470 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:48:48.470 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:48:48.471 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:48:48.471 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:48:48.471 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:48:48.471 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:48:48.472 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:48:48.472 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:48:48.472 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:48.473 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:48.473 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:48:48.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:48.473 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:48:48.473 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:48:48.474 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:48:48.474 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:48.474 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:48:48.476 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:48:53.477 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:48:53.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:48:53.479 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:48:53.519 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:48:53.520 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:48:53.783 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:48:53.784 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:48:53.784 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:48:53.784 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:48:53.784 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:48:53.785 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:48:53.785 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:48:53.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:53.785 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:48:53.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:48:53.785 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:48:53.785 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:48:53.785 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:48:53.786 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:48:53.786 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:48:53.786 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:48:53.786 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:48:53.786 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:48:53.786 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:48:53.787 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:53.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:53.787 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:48:53.787 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:48:53.787 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:48:53.787 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:48:53.787 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:48:53.787 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:48:53.788 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:48:53.788 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:48:53.788 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:48:53.788 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:48:53.788 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:53.788 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:53.788 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:48:53.789 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:48:53.789 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:48:53.789 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:48:53.789 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:48:53.789 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:48:53.789 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:48:53.789 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:48:53.790 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:48:53.790 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:48:53.790 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:53.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:53.790 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:48:53.790 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:48:53.790 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:48:53.790 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:48:53.791 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:48:53.791 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:48:53.791 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:48:53.791 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:48:53.791 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:48:53.791 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:48:53.791 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:53.792 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:53.792 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:48:53.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:53.792 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:48:53.792 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:48:53.792 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:48:53.794 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:53.794 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:48:53.795 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:48:58.796 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:48:58.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:48:58.799 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:48:58.838 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:48:58.839 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:48:59.210 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:48:59.210 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:48:59.210 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:48:59.211 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:48:59.211 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:48:59.211 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:48:59.211 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:48:59.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:59.211 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:48:59.211 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:48:59.212 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:48:59.212 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:48:59.212 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:48:59.212 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:48:59.212 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:48:59.212 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:48:59.212 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:48:59.213 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:48:59.213 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:48:59.213 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:59.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:59.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:48:59.213 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:48:59.213 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:48:59.214 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:48:59.214 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:48:59.214 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:48:59.214 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:48:59.214 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:48:59.214 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:48:59.214 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:48:59.215 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:59.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:59.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:48:59.215 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:48:59.215 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:48:59.215 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:48:59.215 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:48:59.215 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:48:59.216 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:48:59.216 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:48:59.216 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:48:59.216 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:48:59.216 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:59.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:59.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:48:59.217 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:48:59.217 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:48:59.217 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:48:59.217 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:48:59.217 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:48:59.217 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:48:59.217 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:48:59.218 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:48:59.218 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:48:59.218 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:59.218 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:59.218 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:48:59.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:48:59.218 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:48:59.218 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:48:59.219 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:48:59.219 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:48:59.219 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:48:59.220 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:49:04.221 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:49:04.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:49:04.223 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:49:04.262 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:49:04.263 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:49:04.629 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:49:04.629 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:49:04.629 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:49:04.630 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:49:04.630 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:49:04.630 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:49:04.630 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:49:04.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:04.630 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:49:04.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:49:04.631 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:49:04.631 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:49:04.631 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:49:04.631 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:49:04.631 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:49:04.632 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:49:04.632 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:49:04.632 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:49:04.632 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:49:04.632 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:04.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:04.633 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:49:04.633 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:49:04.633 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:49:04.633 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:49:04.633 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:49:04.633 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:49:04.633 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:49:04.634 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:49:04.634 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:49:04.634 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:49:04.634 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:04.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:04.634 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:49:04.634 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:49:04.635 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:49:04.635 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:49:04.636 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:49:04.636 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:49:04.636 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:49:04.636 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:49:04.637 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:49:04.637 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:49:04.637 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:04.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:04.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:49:04.637 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:49:04.638 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:49:04.638 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:49:04.638 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:49:04.638 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:49:04.638 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:49:04.638 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:49:04.639 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:49:04.639 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:49:04.639 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:04.639 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:04.639 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:49:04.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:04.639 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:49:04.640 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:49:04.640 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:49:04.640 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:04.640 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:49:04.641 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:49:09.642 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:49:09.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:49:09.644 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:49:09.683 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:49:09.684 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:49:10.045 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:49:10.045 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:49:10.046 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:49:10.046 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:49:10.046 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:49:10.046 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:49:10.046 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:49:10.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:10.047 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:49:10.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:49:10.047 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:49:10.047 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:49:10.047 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:49:10.047 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:49:10.048 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:49:10.048 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:49:10.048 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:49:10.048 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:49:10.048 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:49:10.048 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:10.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:10.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:49:10.049 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:49:10.049 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:49:10.049 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:49:10.049 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:49:10.050 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:49:10.050 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:49:10.050 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:49:10.050 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:49:10.050 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:49:10.050 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:10.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:10.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:49:10.051 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:49:10.051 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:49:10.051 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:49:10.051 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:49:10.051 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:49:10.052 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:49:10.052 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:49:10.052 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:49:10.052 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:49:10.052 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:10.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:10.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:49:10.052 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:49:10.053 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:49:10.053 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:49:10.053 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:49:10.053 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:49:10.053 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:49:10.053 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:49:10.053 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:49:10.054 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:49:10.054 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:10.054 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:10.054 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:49:10.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:10.054 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:49:10.054 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:49:10.055 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:49:10.055 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:10.055 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:49:10.056 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:49:15.057 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:49:15.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:49:15.059 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:49:15.103 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:49:15.103 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:49:15.468 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:49:15.468 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:49:15.468 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:49:15.469 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:49:15.469 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:49:15.469 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:49:15.470 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:49:15.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:15.470 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:49:15.471 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:49:15.471 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:49:15.471 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:49:15.471 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:49:15.472 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:49:15.472 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:49:15.472 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:49:15.473 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:49:15.473 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:49:15.473 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:49:15.473 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:15.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:15.474 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:49:15.474 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:49:15.474 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:49:15.475 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:49:15.475 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:49:15.475 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:49:15.476 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:49:15.476 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:49:15.476 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:49:15.476 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:49:15.477 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:15.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:15.477 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:49:15.478 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:49:15.478 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:49:15.478 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:49:15.478 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:49:15.479 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:49:15.479 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:49:15.479 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:49:15.480 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:49:15.480 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:49:15.480 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:15.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:15.481 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:49:15.481 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:49:15.481 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:49:15.482 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:49:15.482 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:49:15.482 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:49:15.482 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:49:15.483 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:49:15.483 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:49:15.483 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:49:15.484 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:15.484 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:15.484 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:49:15.484 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:15.485 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:49:15.485 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:49:15.485 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:49:15.486 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:15.486 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:49:15.487 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:49:20.489 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:49:20.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:49:20.491 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:49:20.530 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:49:20.531 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:49:20.899 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:49:20.899 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:49:20.900 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:49:20.900 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:49:20.900 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:49:20.900 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:49:20.900 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:49:20.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:20.900 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:49:20.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:49:20.901 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:49:20.901 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:49:20.901 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:49:20.901 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:49:20.901 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:49:20.901 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:49:20.902 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:49:20.902 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:49:20.902 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:49:20.902 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:20.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:20.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:49:20.902 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:49:20.903 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:49:20.903 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:49:20.903 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:49:20.903 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:49:20.903 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:49:20.903 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:49:20.903 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:49:20.903 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:49:20.904 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:20.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:20.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:49:20.904 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:49:20.904 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:49:20.904 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:49:20.904 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:49:20.905 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:49:20.905 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:49:20.905 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:49:20.905 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:49:20.905 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:49:20.905 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:20.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:20.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:49:20.906 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:49:20.906 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:49:20.906 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:49:20.906 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:49:20.906 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:49:20.906 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:49:20.907 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:49:20.907 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:49:20.907 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:49:20.907 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:20.907 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:20.907 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:49:20.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:20.908 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:49:20.908 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:49:20.908 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:49:20.908 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:20.908 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:49:20.909 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:49:25.911 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:49:25.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:49:25.912 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:49:25.954 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:49:25.954 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:49:26.318 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:49:26.319 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:49:26.319 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:49:26.319 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:49:26.319 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:49:26.319 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:49:26.320 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:49:26.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:26.320 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:49:26.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:49:26.320 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:49:26.320 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:49:26.320 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:49:26.321 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:49:26.321 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:49:26.321 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:49:26.321 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:49:26.321 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:49:26.321 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:49:26.322 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:26.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:26.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:49:26.322 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:49:26.322 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:49:26.322 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:49:26.322 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:49:26.323 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:49:26.323 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:49:26.323 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:49:26.323 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:49:26.323 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:49:26.324 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:26.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:26.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:49:26.324 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:49:26.324 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:49:26.324 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:49:26.324 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:49:26.325 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:49:26.325 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:49:26.325 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:49:26.325 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:49:26.325 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:49:26.325 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:26.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:26.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:49:26.326 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:49:26.326 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:49:26.326 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:49:26.326 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:49:26.326 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:49:26.327 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:49:26.327 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:49:26.327 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:49:26.327 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:49:26.327 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:26.327 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:26.327 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:49:26.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:26.328 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:49:26.328 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:49:26.328 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:49:26.328 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:26.328 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:49:26.329 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:49:31.330 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:49:31.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:49:31.332 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:49:31.373 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:49:31.374 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:49:31.739 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:49:31.739 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:49:31.739 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:49:31.739 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:49:31.739 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:49:31.740 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:49:31.740 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:49:31.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:31.740 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:49:31.740 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:49:31.740 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:49:31.740 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:49:31.741 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:49:31.741 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:49:31.741 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:49:31.741 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:49:31.741 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:49:31.741 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:49:31.742 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:49:31.742 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:31.742 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:31.742 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:49:31.742 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:49:31.742 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:49:31.743 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:49:31.743 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:49:31.743 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:49:31.743 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:49:31.743 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:49:31.743 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:49:31.743 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:49:31.744 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:31.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:31.744 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:49:31.744 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:49:31.744 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:49:31.744 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:49:31.745 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:49:31.745 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:49:31.745 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:49:31.745 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:49:31.745 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:49:31.745 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:49:31.745 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:31.746 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:31.746 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:49:31.746 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:49:31.746 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:49:31.746 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:49:31.746 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:49:31.747 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:49:31.747 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:49:31.747 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:49:31.747 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:49:31.747 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:49:31.747 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:31.747 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:31.748 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:49:31.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:31.748 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:49:31.748 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:49:31.748 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:49:31.748 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:31.749 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:49:31.751 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:49:36.752 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:49:36.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:49:36.754 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:49:36.794 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:49:36.794 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:49:37.154 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:49:37.155 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:49:37.155 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:49:37.155 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:49:37.155 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:49:37.155 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:49:37.155 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:49:37.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:37.156 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:49:37.156 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:49:37.156 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:49:37.156 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:49:37.156 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:49:37.156 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:49:37.156 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:49:37.157 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:49:37.157 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:49:37.157 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:49:37.157 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:49:37.157 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:37.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:37.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:49:37.160 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:49:37.160 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:49:37.160 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:49:37.160 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:49:37.160 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:49:37.160 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:49:37.161 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:49:37.161 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:49:37.161 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:49:37.161 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:37.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:37.161 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:49:37.162 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:49:37.162 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:49:37.162 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:49:37.162 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:49:37.162 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:49:37.162 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:49:37.163 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:49:37.163 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:49:37.163 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:49:37.163 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:37.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:37.163 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:49:37.163 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:49:37.164 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:49:37.164 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:49:37.164 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:49:37.164 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:49:37.164 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:49:37.164 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:49:37.164 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:49:37.165 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:49:37.165 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:37.165 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:37.165 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:49:37.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:37.165 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:49:37.165 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:49:37.166 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:49:37.166 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:37.166 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:49:37.167 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:49:42.168 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:49:42.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:49:42.169 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:49:42.210 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:49:42.211 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:49:42.577 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:49:42.578 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:49:42.578 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:49:42.578 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:49:42.578 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:49:42.578 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:49:42.579 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:49:42.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:42.579 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:49:42.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:49:42.579 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:49:42.579 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:49:42.579 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:49:42.580 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:49:42.580 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:49:42.580 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:49:42.580 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:49:42.580 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:49:42.580 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:49:42.580 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:42.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:42.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:49:42.581 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:49:42.581 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:49:42.581 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:49:42.581 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:49:42.581 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:49:42.581 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:49:42.582 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:49:42.582 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:49:42.582 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:49:42.582 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:42.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:42.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:49:42.582 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:49:42.583 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:49:42.583 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:49:42.583 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:49:42.583 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:49:42.583 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:49:42.583 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:49:42.583 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:49:42.583 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:49:42.584 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:42.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:42.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:49:42.584 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:49:42.584 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:49:42.584 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:49:42.584 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:49:42.585 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:49:42.585 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:49:42.585 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:49:42.585 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:49:42.585 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:49:42.585 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:42.585 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:42.586 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:49:42.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:42.586 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:49:42.586 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:49:42.586 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:49:42.586 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:42.586 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:49:42.587 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:49:47.588 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:49:47.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:49:47.590 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:49:47.631 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:49:47.631 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:49:47.987 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:49:47.988 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:49:47.988 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:49:47.988 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:49:47.988 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:49:47.988 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:49:47.989 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:49:47.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:47.989 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:49:47.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:49:47.989 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:49:47.989 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:49:47.989 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:49:47.990 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:49:47.990 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:49:47.990 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:49:47.990 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:49:47.990 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:49:47.990 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:49:47.990 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:47.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:47.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:49:47.991 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:49:47.991 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:49:47.991 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:49:47.991 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:49:47.991 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:49:47.992 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:49:47.992 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:49:47.992 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:49:47.992 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:49:47.992 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:47.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:47.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:49:47.993 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:49:47.993 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:49:47.993 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:49:47.993 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:49:47.993 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:49:47.993 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:49:47.993 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:49:47.993 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:49:47.994 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:49:47.994 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:47.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:47.994 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:49:47.994 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:49:47.994 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:49:47.994 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:49:47.995 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:49:47.995 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:49:47.995 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:49:47.995 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:49:47.995 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:49:47.995 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:49:47.995 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:47.995 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:47.996 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:49:47.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:47.996 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:49:47.996 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:49:47.996 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:49:47.997 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:47.997 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:49:47.998 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:49:52.999 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:49:52.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:49:53.000 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:49:53.049 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:49:53.049 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:49:53.417 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:49:53.417 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:49:53.417 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:49:53.417 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:49:53.417 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:49:53.418 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:49:53.418 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:49:53.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:53.418 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:49:53.418 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:49:53.418 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:49:53.419 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:49:53.419 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:49:53.419 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:49:53.419 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:49:53.419 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:49:53.419 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:49:53.419 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:49:53.419 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:49:53.420 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:53.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:53.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:49:53.420 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:49:53.420 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:49:53.420 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:49:53.421 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:49:53.421 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:49:53.421 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:49:53.421 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:49:53.422 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:49:53.422 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:49:53.422 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:53.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:53.422 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:49:53.423 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:49:53.423 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:49:53.423 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:49:53.423 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:49:53.423 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:49:53.423 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:49:53.423 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:49:53.424 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:49:53.424 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:49:53.424 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:53.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:53.424 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:49:53.424 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:49:53.425 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:49:53.426 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:49:53.426 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:49:53.426 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:49:53.426 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:49:53.426 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:49:53.426 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:49:53.426 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:49:53.427 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:53.427 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:53.427 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:49:53.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:53.427 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:49:53.427 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:49:53.427 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:49:53.428 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:53.428 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:49:53.429 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:49:58.430 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:49:58.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:49:58.431 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:49:58.472 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:49:58.473 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:49:58.839 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:49:58.839 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:49:58.840 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:49:58.840 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:49:58.840 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:49:58.840 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:49:58.840 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:49:58.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:58.840 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:49:58.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:49:58.841 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:49:58.841 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:49:58.841 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:49:58.841 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:49:58.841 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:49:58.841 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:49:58.842 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:49:58.842 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:49:58.842 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:49:58.842 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:58.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:58.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:49:58.842 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:49:58.843 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:49:58.843 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:49:58.843 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:49:58.843 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:49:58.843 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:49:58.843 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:49:58.843 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:49:58.844 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:49:58.844 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:58.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:58.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:49:58.844 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:49:58.844 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:49:58.844 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:49:58.844 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:49:58.845 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:49:58.845 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:49:58.845 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:49:58.845 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:49:58.845 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:49:58.845 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:58.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:58.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:49:58.846 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:49:58.846 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:49:58.846 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:49:58.846 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:49:58.846 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:49:58.846 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:49:58.847 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:49:58.847 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:49:58.847 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:49:58.847 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:58.847 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:58.847 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:49:58.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:49:58.847 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:49:58.848 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:49:58.848 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:49:58.848 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:49:58.848 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:49:58.849 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:50:03.850 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:50:03.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:50:03.852 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:50:03.891 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:50:03.891 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:50:04.262 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:50:04.262 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:50:04.262 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:50:04.262 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:50:04.263 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:50:04.263 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:50:04.263 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:50:04.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:04.263 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:50:04.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:50:04.263 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:50:04.264 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:50:04.264 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:50:04.264 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:50:04.264 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:50:04.264 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:50:04.264 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:50:04.264 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:50:04.265 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:50:04.265 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:04.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:04.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:50:04.265 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:50:04.265 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:50:04.265 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:50:04.266 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:50:04.266 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:50:04.266 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:50:04.266 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:50:04.266 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:50:04.266 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:50:04.266 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:04.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:04.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:50:04.267 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:50:04.267 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:50:04.267 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:50:04.267 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:50:04.267 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:50:04.267 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:50:04.268 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:50:04.268 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:50:04.268 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:50:04.268 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:04.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:04.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:50:04.268 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:50:04.269 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:50:04.269 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:50:04.269 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:50:04.269 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:50:04.269 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:50:04.269 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:50:04.269 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:50:04.269 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:50:04.270 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:04.270 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:04.270 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:50:04.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:04.270 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:50:04.270 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:50:04.270 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:50:04.271 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:04.271 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:50:04.272 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:50:09.273 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:50:09.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:50:09.274 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:50:09.314 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:50:09.315 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:50:09.682 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:50:09.683 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:50:09.683 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:50:09.683 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:50:09.683 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:50:09.683 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:50:09.684 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:50:09.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:09.684 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:50:09.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:50:09.684 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:50:09.684 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:50:09.684 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:50:09.685 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:50:09.685 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:50:09.685 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:50:09.685 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:50:09.685 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:50:09.685 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:50:09.686 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:09.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:09.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:50:09.686 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:50:09.686 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:50:09.686 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:50:09.686 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:50:09.687 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:50:09.687 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:50:09.687 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:50:09.687 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:50:09.687 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:50:09.687 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:09.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:09.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:50:09.688 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:50:09.688 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:50:09.688 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:50:09.688 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:50:09.688 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:50:09.688 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:50:09.688 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:50:09.689 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:50:09.689 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:50:09.689 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:09.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:09.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:50:09.689 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:50:09.689 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:50:09.690 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:50:09.690 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:50:09.690 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:50:09.690 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:50:09.690 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:50:09.690 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:50:09.690 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:50:09.691 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:09.691 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:09.691 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:50:09.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:09.691 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:50:09.691 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:50:09.691 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:50:09.692 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:09.692 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:50:09.693 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:50:14.694 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:50:14.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:50:14.695 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:50:14.736 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:50:14.736 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:50:15.099 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:50:15.099 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:50:15.099 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:50:15.100 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:50:15.100 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:50:15.100 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:50:15.100 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:50:15.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:15.100 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:50:15.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:50:15.101 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:50:15.101 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:50:15.101 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:50:15.101 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:50:15.101 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:50:15.102 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:50:15.102 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:50:15.102 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:50:15.102 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:50:15.102 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:15.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:15.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:50:15.103 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:50:15.103 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:50:15.103 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:50:15.103 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:50:15.103 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:50:15.103 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:50:15.104 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:50:15.104 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:50:15.104 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:50:15.104 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:15.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:15.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:50:15.104 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:50:15.105 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:50:15.105 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:50:15.105 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:50:15.105 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:50:15.105 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:50:15.105 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:50:15.106 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:50:15.106 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:50:15.106 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:15.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:15.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:50:15.106 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:50:15.106 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:50:15.107 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:50:15.107 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:50:15.107 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:50:15.107 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:50:15.107 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:50:15.107 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:50:15.107 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:50:15.108 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:15.108 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:15.108 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:50:15.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:15.108 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:50:15.109 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:50:15.109 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:50:15.109 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:15.109 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:50:15.110 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:50:20.111 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:50:20.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:50:20.113 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:50:20.155 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:50:20.155 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:50:20.524 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:50:20.525 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:50:20.525 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:50:20.525 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:50:20.525 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:50:20.526 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:50:20.526 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:50:20.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:20.526 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:50:20.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:50:20.526 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:50:20.526 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:50:20.527 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:50:20.527 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:50:20.527 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:50:20.527 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:50:20.527 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:50:20.527 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:50:20.527 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:50:20.528 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:20.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:20.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:50:20.528 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:50:20.528 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:50:20.528 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:50:20.528 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:50:20.529 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:50:20.529 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:50:20.529 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:50:20.529 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:50:20.529 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:50:20.529 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:20.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:20.530 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:50:20.530 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:50:20.530 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:50:20.530 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:50:20.530 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:50:20.530 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:50:20.530 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:50:20.530 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:50:20.531 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:50:20.531 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:50:20.531 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:20.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:20.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:50:20.531 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:50:20.532 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:50:20.532 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:50:20.532 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:50:20.532 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:50:20.532 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:50:20.532 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:50:20.532 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:50:20.533 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:50:20.533 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:20.533 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:20.533 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:50:20.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:20.533 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:50:20.533 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:50:20.533 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:50:20.534 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:20.534 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:50:20.535 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:50:25.537 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:50:25.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:50:25.539 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:50:25.582 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:50:25.583 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:50:25.941 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:50:25.941 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:50:25.941 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:50:25.941 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:50:25.941 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:50:25.942 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:50:25.942 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:50:25.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:25.942 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:50:25.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:50:25.942 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:50:25.943 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:50:25.943 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:50:25.943 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:50:25.943 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:50:25.943 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:50:25.943 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:50:25.943 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:50:25.944 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:50:25.944 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:25.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:25.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:50:25.944 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:50:25.944 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:50:25.945 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:50:25.945 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:50:25.945 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:50:25.945 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:50:25.945 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:50:25.945 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:50:25.945 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:50:25.946 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:25.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:25.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:50:25.946 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:50:25.946 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:50:25.946 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:50:25.946 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:50:25.947 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:50:25.947 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:50:25.947 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:50:25.947 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:50:25.947 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:50:25.947 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:25.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:25.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:50:25.948 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:50:25.948 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:50:25.948 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:50:25.948 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:50:25.948 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:50:25.948 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:50:25.948 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:50:25.949 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:50:25.949 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:50:25.949 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:25.949 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:25.949 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:50:25.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:25.949 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:50:25.950 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:50:25.950 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:50:25.950 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:25.950 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:50:25.951 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:50:30.952 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:50:30.953 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:50:30.954 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:50:30.995 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:50:30.995 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:50:31.361 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:50:31.362 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:50:31.362 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:50:31.362 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:50:31.362 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:50:31.363 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:50:31.363 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:50:31.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:31.363 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:50:31.363 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:50:31.363 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:50:31.363 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:50:31.364 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:50:31.364 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:50:31.364 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:50:31.364 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:50:31.364 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:50:31.364 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:50:31.365 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:50:31.365 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:31.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:31.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:50:31.365 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:50:31.365 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:50:31.365 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:50:31.366 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:50:31.366 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:50:31.366 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:50:31.366 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:50:31.366 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:50:31.366 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:50:31.366 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:31.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:31.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:50:31.367 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:50:31.367 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:50:31.367 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:50:31.367 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:50:31.368 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:50:31.368 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:50:31.368 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:50:31.368 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:50:31.368 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:50:31.368 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:31.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:31.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:50:31.369 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:50:31.369 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:50:31.369 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:50:31.369 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:50:31.369 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:50:31.370 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:50:31.370 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:50:31.370 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:50:31.370 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:50:31.370 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:31.370 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:31.370 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:50:31.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:31.371 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:50:31.371 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:50:31.371 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:50:31.371 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:31.371 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:50:31.372 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:50:36.374 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:50:36.374 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:50:36.375 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:50:36.416 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:50:36.416 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:50:36.787 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:50:36.787 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:50:36.787 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:50:36.787 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:50:36.787 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:50:36.788 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:50:36.788 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:50:36.788 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:36.788 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:50:36.788 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:50:36.788 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:50:36.788 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:50:36.789 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:50:36.789 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:50:36.789 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:50:36.789 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:50:36.789 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:50:36.789 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:50:36.789 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:50:36.790 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:36.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:36.790 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:50:36.790 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:50:36.790 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:50:36.790 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:50:36.790 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:50:36.791 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:50:36.791 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:50:36.791 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:50:36.791 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:50:36.791 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:50:36.791 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:36.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:36.792 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:50:36.792 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:50:36.792 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:50:36.792 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:50:36.792 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:50:36.792 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:50:36.792 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:50:36.793 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:50:36.793 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:50:36.793 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:50:36.793 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:36.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:36.793 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:50:36.793 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:50:36.794 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:50:36.794 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:50:36.794 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:50:36.794 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:50:36.794 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:50:36.794 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:50:36.794 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:50:36.795 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:50:36.795 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:36.795 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:36.795 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:50:36.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:36.795 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:50:36.795 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:50:36.795 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:50:36.796 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:36.796 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:50:36.797 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:50:41.798 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:50:41.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:50:41.799 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:50:41.840 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:50:41.841 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:50:42.207 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:50:42.208 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:50:42.208 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:50:42.208 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:50:42.209 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:50:42.209 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:50:42.209 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:50:42.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:42.210 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:50:42.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:50:42.210 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:50:42.211 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:50:42.211 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:50:42.211 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:50:42.211 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:50:42.212 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:50:42.212 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:50:42.212 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:50:42.212 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:50:42.212 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:42.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:42.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:50:42.213 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:50:42.213 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:50:42.213 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:50:42.213 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:50:42.214 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:50:42.214 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:50:42.214 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:50:42.214 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:50:42.214 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:50:42.214 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:42.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:42.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:50:42.215 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:50:42.215 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:50:42.215 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:50:42.215 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:50:42.215 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:50:42.215 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:50:42.215 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:50:42.216 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:50:42.216 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:50:42.216 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:42.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:42.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:50:42.216 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:50:42.216 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:50:42.217 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:50:42.217 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:50:42.217 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:50:42.217 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:50:42.217 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:50:42.217 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:50:42.217 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:50:42.218 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:42.218 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:42.218 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:50:42.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:42.218 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:50:42.219 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:50:42.219 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:50:42.219 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:42.219 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:50:42.220 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:50:47.222 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:50:47.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:50:47.223 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:50:47.264 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:50:47.264 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:50:47.635 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:50:47.635 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:50:47.635 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:50:47.635 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:50:47.635 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:50:47.636 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:50:47.636 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:50:47.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:47.636 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:50:47.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:50:47.636 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:50:47.636 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:50:47.637 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:50:47.637 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:50:47.637 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:50:47.637 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:50:47.637 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:50:47.637 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:50:47.637 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:50:47.638 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:47.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:47.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:50:47.638 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:50:47.638 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:50:47.638 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:50:47.638 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:50:47.639 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:50:47.639 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:50:47.639 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:50:47.639 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:50:47.639 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:50:47.639 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:47.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:47.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:50:47.640 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:50:47.640 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:50:47.640 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:50:47.640 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:50:47.640 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:50:47.640 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:50:47.641 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:50:47.641 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:50:47.641 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:50:47.641 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:47.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:47.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:50:47.641 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:50:47.642 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:50:47.642 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:50:47.642 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:50:47.642 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:50:47.642 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:50:47.642 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:50:47.642 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:50:47.643 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:50:47.643 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:47.643 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:47.643 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:50:47.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:47.643 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:50:47.643 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:50:47.644 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:50:47.644 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:47.644 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:50:47.645 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:50:52.646 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:50:52.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:50:52.647 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:50:52.686 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:50:52.686 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:50:53.051 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:50:53.051 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:50:53.051 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:50:53.052 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:50:53.052 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:50:53.052 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:50:53.052 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:50:53.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:53.053 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:50:53.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:50:53.053 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:50:53.053 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:50:53.053 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:50:53.053 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:50:53.054 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:50:53.054 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:50:53.054 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:50:53.054 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:50:53.054 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:50:53.054 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:53.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:53.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:50:53.055 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:50:53.055 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:50:53.055 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:50:53.055 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:50:53.055 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:50:53.056 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:50:53.056 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:50:53.056 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:50:53.056 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:50:53.056 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:53.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:53.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:50:53.057 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:50:53.057 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:50:53.057 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:50:53.057 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:50:53.057 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:50:53.057 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:50:53.058 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:50:53.058 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:50:53.058 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:50:53.058 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:53.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:53.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:50:53.059 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:50:53.059 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:50:53.059 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:50:53.059 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:50:53.059 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:50:53.059 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:50:53.059 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:50:53.060 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:50:53.060 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:50:53.060 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:53.060 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:53.060 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:50:53.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:53.060 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:50:53.060 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:50:53.061 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:50:53.061 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:53.061 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:50:53.062 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:50:58.064 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:50:58.065 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:50:58.066 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:50:58.107 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:50:58.107 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:50:58.471 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:50:58.472 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:50:58.472 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:50:58.472 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:50:58.472 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:50:58.472 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:50:58.472 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:50:58.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:58.473 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:50:58.473 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:50:58.473 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:50:58.473 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:50:58.473 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:50:58.474 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:50:58.474 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:50:58.474 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:50:58.474 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:50:58.474 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:50:58.474 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:50:58.474 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:58.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:58.475 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:50:58.475 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:50:58.475 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:50:58.475 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:50:58.475 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:50:58.476 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:50:58.476 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:50:58.476 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:50:58.476 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:50:58.476 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:50:58.476 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:58.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:58.477 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:50:58.477 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:50:58.477 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:50:58.477 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:50:58.477 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:50:58.477 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:50:58.477 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:50:58.478 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:50:58.478 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:50:58.478 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:50:58.478 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:58.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:58.478 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:50:58.478 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:50:58.479 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:50:58.479 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:50:58.479 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:50:58.479 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:50:58.479 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:50:58.479 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:50:58.480 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:50:58.480 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:50:58.480 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:58.480 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:58.480 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:50:58.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:50:58.480 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:50:58.480 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:50:58.481 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:50:58.481 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:50:58.481 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:50:58.482 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:51:03.484 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:51:03.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:51:03.485 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:51:03.526 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:51:03.526 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:51:03.889 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:51:03.889 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:51:03.889 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:51:03.890 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:51:03.890 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:51:03.890 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:51:03.890 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:51:03.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:03.890 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:51:03.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:51:03.891 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:51:03.891 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:51:03.891 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:51:03.891 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:51:03.891 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:51:03.891 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:51:03.892 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:51:03.892 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:51:03.892 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:51:03.892 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:03.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:03.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:51:03.892 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:51:03.893 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:51:03.893 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:51:03.893 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:51:03.893 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:51:03.893 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:51:03.893 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:51:03.893 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:51:03.893 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:51:03.894 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:03.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:03.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:51:03.894 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:51:03.894 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:51:03.894 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:51:03.894 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:51:03.895 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:51:03.895 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:51:03.895 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:51:03.895 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:51:03.895 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:51:03.895 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:03.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:03.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:51:03.896 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:51:03.896 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:51:03.896 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:51:03.896 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:51:03.896 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:51:03.896 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:51:03.897 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:51:03.897 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:51:03.897 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:51:03.897 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:03.897 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:03.897 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:51:03.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:03.897 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:51:03.898 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:51:03.898 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:51:03.898 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:03.898 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:51:03.899 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:51:08.900 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:51:08.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:51:08.901 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:51:08.942 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:51:08.943 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:51:09.306 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:51:09.307 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:51:09.307 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:51:09.307 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:51:09.307 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:51:09.307 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:51:09.308 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:51:09.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:09.308 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:51:09.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:51:09.308 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:51:09.308 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:51:09.308 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:51:09.309 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:51:09.309 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:51:09.309 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:51:09.309 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:51:09.309 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:51:09.309 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:51:09.309 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:09.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:09.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:51:09.310 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:51:09.310 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:51:09.310 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:51:09.310 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:51:09.310 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:51:09.311 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:51:09.311 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:51:09.311 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:51:09.311 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:51:09.311 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:09.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:09.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:51:09.311 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:51:09.312 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:51:09.312 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:51:09.312 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:51:09.312 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:51:09.312 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:51:09.312 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:51:09.312 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:51:09.313 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:51:09.313 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:09.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:09.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:51:09.313 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:51:09.313 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:51:09.313 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:51:09.313 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:51:09.314 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:51:09.314 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:51:09.314 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:51:09.314 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:51:09.314 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:51:09.314 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:09.315 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:09.315 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:51:09.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:09.315 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:51:09.315 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:51:09.315 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:51:09.315 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:09.316 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:51:09.316 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:51:14.318 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:51:14.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:51:14.319 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:51:14.358 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:51:14.359 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:51:14.732 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:51:14.733 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:51:14.733 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:51:14.733 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:51:14.733 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:51:14.733 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:51:14.734 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:51:14.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:14.734 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:51:14.734 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:51:14.734 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:51:14.734 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:51:14.735 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:51:14.735 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:51:14.735 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:51:14.735 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:51:14.735 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:51:14.735 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:51:14.735 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:51:14.736 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:14.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:14.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:51:14.736 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:51:14.736 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:51:14.737 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:51:14.737 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:51:14.737 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:51:14.737 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:51:14.737 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:51:14.737 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:51:14.737 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:51:14.738 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:14.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:14.738 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:51:14.738 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:51:14.738 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:51:14.738 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:51:14.739 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:51:14.739 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:51:14.739 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:51:14.739 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:51:14.739 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:51:14.739 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:51:14.740 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:14.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:14.740 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:51:14.741 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:51:14.741 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:51:14.741 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:51:14.741 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:51:14.741 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:51:14.741 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:51:14.742 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:51:14.742 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:51:14.742 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:51:14.742 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:14.742 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:14.743 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:51:14.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:14.743 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:51:14.743 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:51:14.743 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:51:14.743 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:14.743 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:51:14.744 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:51:19.746 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:51:19.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:51:19.747 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:51:19.787 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:51:19.787 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:51:20.162 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:51:20.163 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:51:20.163 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:51:20.163 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:51:20.163 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:51:20.163 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:51:20.164 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:51:20.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:20.164 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:51:20.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:51:20.164 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:51:20.164 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:51:20.165 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:51:20.165 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:51:20.165 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:51:20.165 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:51:20.165 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:51:20.166 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:51:20.166 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:51:20.166 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:20.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:20.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:51:20.166 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:51:20.166 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:51:20.167 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:51:20.167 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:51:20.167 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:51:20.167 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:51:20.167 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:51:20.167 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:51:20.167 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:51:20.168 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:20.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:20.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:51:20.168 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:51:20.168 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:51:20.168 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:51:20.168 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:51:20.169 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:51:20.169 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:51:20.169 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:51:20.169 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:51:20.169 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:51:20.169 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:20.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:20.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:51:20.170 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:51:20.170 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:51:20.170 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:51:20.170 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:51:20.170 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:51:20.170 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:51:20.170 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:51:20.171 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:51:20.171 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:51:20.171 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:20.171 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:20.171 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:51:20.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:20.171 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:51:20.172 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:51:20.172 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:51:20.172 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:20.172 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:51:20.173 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:51:25.175 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:51:25.176 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:51:25.177 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:51:25.226 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:51:25.226 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:51:25.600 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:51:25.600 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:51:25.601 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:51:25.601 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:51:25.601 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:51:25.601 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:51:25.602 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:51:25.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:25.602 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:51:25.603 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:51:25.603 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:51:25.603 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:51:25.604 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:51:25.604 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:51:25.604 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:51:25.604 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:51:25.604 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:51:25.605 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:51:25.605 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:51:25.605 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:25.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:25.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:51:25.606 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:51:25.606 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:51:25.606 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:51:25.607 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:51:25.607 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:51:25.607 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:51:25.608 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:51:25.608 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:51:25.608 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:51:25.608 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:25.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:25.609 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:51:25.609 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:51:25.609 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:51:25.609 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:51:25.610 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:51:25.610 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:51:25.610 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:51:25.610 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:51:25.611 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:51:25.611 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:51:25.611 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:25.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:25.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:51:25.612 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:51:25.612 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:51:25.612 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:51:25.613 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:51:25.613 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:51:25.613 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:51:25.613 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:51:25.614 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:51:25.614 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:51:25.614 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:25.614 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:25.615 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:51:25.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:25.615 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:51:25.615 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:51:25.616 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:51:25.616 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:25.616 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:51:25.618 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:51:30.619 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:51:30.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:51:30.621 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:51:30.659 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:51:30.659 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:51:31.022 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:51:31.022 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:51:31.022 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:51:31.022 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:51:31.022 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:51:31.023 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:51:31.023 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:51:31.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:31.023 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:51:31.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:51:31.023 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:51:31.024 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:51:31.024 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:51:31.024 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:51:31.024 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:51:31.024 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:51:31.024 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:51:31.024 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:51:31.025 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:51:31.025 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:31.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:31.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:51:31.025 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:51:31.025 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:51:31.025 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:51:31.025 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:51:31.026 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:51:31.026 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:51:31.026 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:51:31.026 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:51:31.026 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:51:31.026 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:31.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:31.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:51:31.027 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:51:31.027 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:51:31.027 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:51:31.027 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:51:31.028 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:51:31.028 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:51:31.028 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:51:31.028 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:51:31.028 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:51:31.028 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:31.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:31.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:51:31.029 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:51:31.029 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:51:31.029 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:51:31.029 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:51:31.029 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:51:31.029 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:51:31.029 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:51:31.030 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:51:31.030 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:51:31.030 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:31.030 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:31.031 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:51:31.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:31.031 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:51:31.031 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:51:31.031 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:51:31.031 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:31.031 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:51:31.032 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:51:36.034 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:51:36.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:51:36.035 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:51:36.075 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:51:36.075 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:51:36.450 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:51:36.450 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:51:36.450 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:51:36.450 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:51:36.450 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:51:36.451 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:51:36.451 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:51:36.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:36.451 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:51:36.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:51:36.451 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:51:36.451 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:51:36.452 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:51:36.452 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:51:36.452 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:51:36.452 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:51:36.452 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:51:36.452 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:51:36.452 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:51:36.453 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:36.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:36.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:51:36.453 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:51:36.453 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:51:36.453 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:51:36.453 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:51:36.453 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:51:36.454 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:51:36.454 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:51:36.454 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:51:36.454 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:51:36.454 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:36.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:36.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:51:36.455 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:51:36.455 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:51:36.455 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:51:36.455 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:51:36.455 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:51:36.455 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:51:36.455 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:51:36.456 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:51:36.456 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:51:36.456 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:36.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:36.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:51:36.456 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:51:36.456 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:51:36.457 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:51:36.457 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:51:36.457 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:51:36.457 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:51:36.457 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:51:36.457 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:51:36.457 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:51:36.457 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:36.458 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:36.458 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:51:36.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:36.458 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:51:36.458 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:51:36.458 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:51:36.458 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:36.459 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:51:36.459 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:51:41.461 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:51:41.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:51:41.463 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:51:41.503 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:51:41.503 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:51:41.869 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:51:41.870 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:51:41.870 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:51:41.870 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:51:41.870 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:51:41.870 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:51:41.871 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:51:41.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:41.871 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:51:41.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:51:41.871 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:51:41.871 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:51:41.871 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:51:41.872 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:51:41.872 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:51:41.872 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:51:41.872 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:51:41.872 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:51:41.872 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:51:41.872 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:41.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:41.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:51:41.873 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:51:41.873 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:51:41.873 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:51:41.873 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:51:41.873 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:51:41.873 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:51:41.874 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:51:41.874 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:51:41.874 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:51:41.874 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:41.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:41.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:51:41.875 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:51:41.875 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:51:41.875 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:51:41.875 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:51:41.875 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:51:41.875 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:51:41.875 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:51:41.875 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:51:41.876 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:51:41.876 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:41.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:41.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:51:41.876 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:51:41.876 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:51:41.876 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:51:41.877 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:51:41.877 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:51:41.877 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:51:41.877 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:51:41.877 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:51:41.877 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:51:41.877 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:41.877 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:41.878 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:51:41.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:41.878 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:51:41.878 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:51:41.878 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:51:41.878 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:41.878 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:51:41.879 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:51:46.880 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:51:46.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:51:46.882 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:51:46.922 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:51:46.923 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:51:47.301 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:51:47.301 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:51:47.301 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:51:47.301 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:51:47.301 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:51:47.302 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:51:47.302 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:51:47.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:47.302 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:51:47.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:51:47.302 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:51:47.303 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:51:47.303 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:51:47.303 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:51:47.303 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:51:47.303 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:51:47.303 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:51:47.303 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:51:47.304 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:51:47.304 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:47.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:47.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:51:47.304 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:51:47.305 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:51:47.305 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:51:47.305 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:51:47.305 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:51:47.305 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:51:47.305 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:51:47.305 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:51:47.306 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:51:47.306 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:47.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:47.306 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:51:47.306 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:51:47.306 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:51:47.306 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:51:47.307 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:51:47.307 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:51:47.307 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:51:47.307 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:51:47.307 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:51:47.307 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:51:47.307 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:47.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:47.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:51:47.308 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:51:47.308 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:51:47.308 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:51:47.308 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:51:47.308 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:51:47.309 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:51:47.309 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:51:47.309 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:51:47.309 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:51:47.309 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:47.309 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:47.309 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:51:47.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:47.310 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:51:47.310 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:51:47.310 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:51:47.310 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:47.310 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:51:47.311 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:51:52.313 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:51:52.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:51:52.314 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:51:52.356 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:51:52.356 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:51:52.727 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:51:52.727 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:51:52.728 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:51:52.728 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:51:52.728 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:51:52.728 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:51:52.728 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:51:52.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:52.728 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:51:52.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:51:52.729 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:51:52.729 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:51:52.729 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:51:52.729 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:51:52.729 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:51:52.729 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:51:52.730 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:51:52.730 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:51:52.730 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:51:52.730 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:52.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:52.730 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:51:52.730 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:51:52.731 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:51:52.731 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:51:52.731 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:51:52.731 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:51:52.731 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:51:52.731 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:51:52.731 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:51:52.731 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:51:52.732 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:52.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:52.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:51:52.732 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:51:52.732 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:51:52.732 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:51:52.732 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:51:52.733 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:51:52.733 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:51:52.733 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:51:52.733 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:51:52.733 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:51:52.733 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:52.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:52.733 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:51:52.734 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:51:52.734 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:51:52.734 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:51:52.734 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:51:52.734 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:51:52.734 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:51:52.734 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:51:52.735 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:51:52.735 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:51:52.735 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:52.735 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:52.735 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:51:52.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:52.735 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:51:52.742 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:51:52.742 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:51:52.742 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:52.742 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:51:52.744 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:51:57.745 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:51:57.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:51:57.746 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:51:57.786 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:51:57.786 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:51:58.158 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:51:58.158 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:51:58.158 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:51:58.158 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:51:58.159 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:51:58.159 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:51:58.159 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:51:58.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:58.159 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:51:58.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:51:58.160 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:51:58.160 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:51:58.160 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:51:58.160 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:51:58.161 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:51:58.161 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:51:58.161 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:51:58.161 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:51:58.161 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:51:58.162 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:58.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:58.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:51:58.162 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:51:58.162 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:51:58.162 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:51:58.163 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:51:58.163 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:51:58.163 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:51:58.163 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:51:58.163 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:51:58.164 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:51:58.164 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:58.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:58.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:51:58.164 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:51:58.165 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:51:58.165 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:51:58.165 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:51:58.165 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:51:58.165 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:51:58.166 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:51:58.166 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:51:58.166 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:51:58.166 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:58.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:58.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:51:58.167 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:51:58.167 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:51:58.167 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:51:58.167 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:51:58.167 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:51:58.168 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:51:58.168 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:51:58.168 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:51:58.168 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:51:58.168 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:58.169 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:58.169 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:51:58.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:51:58.169 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:51:58.169 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:51:58.169 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:51:58.170 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:51:58.170 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:51:58.171 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:52:03.173 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:52:03.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:52:03.174 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:52:03.214 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:52:03.215 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:52:03.578 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:52:03.578 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:52:03.578 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:52:03.578 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:52:03.579 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:52:03.579 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:52:03.579 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:52:03.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:03.579 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:52:03.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:03.579 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:52:03.580 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:03.580 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:03.580 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:52:03.580 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:03.580 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:52:03.580 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:03.580 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:52:03.581 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:52:03.581 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:03.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:03.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:03.581 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:52:03.581 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:03.581 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:03.581 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:52:03.582 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:03.582 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:52:03.582 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:03.582 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:52:03.582 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:52:03.582 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:03.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:03.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:03.583 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:52:03.583 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:03.583 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:03.583 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:52:03.583 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:03.583 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:52:03.584 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:03.584 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:52:03.584 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:52:03.584 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:03.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:03.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:03.584 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:52:03.585 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:03.585 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:03.585 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:52:03.585 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:03.585 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:52:03.585 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:03.585 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:52:03.585 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:52:03.586 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:03.586 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:03.586 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:52:03.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:03.586 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:52:03.586 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:52:03.586 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:52:03.587 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:03.587 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:52:03.588 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:52:08.589 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:52:08.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:52:08.590 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:52:08.631 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:52:08.632 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:52:09.001 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:52:09.002 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:52:09.002 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:52:09.002 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:52:09.002 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:52:09.002 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:52:09.002 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:52:09.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:09.003 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:52:09.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:09.003 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:52:09.003 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:09.003 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:09.003 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:52:09.004 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:09.004 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:52:09.004 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:09.004 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:52:09.004 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:52:09.004 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:09.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:09.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:09.005 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:52:09.005 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:09.005 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:09.005 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:52:09.005 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:09.005 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:52:09.006 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:09.006 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:52:09.006 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:52:09.006 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:09.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:09.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:09.006 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:52:09.007 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:09.007 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:09.007 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:52:09.007 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:09.007 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:52:09.007 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:09.007 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:52:09.007 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:52:09.008 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:09.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:09.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:09.008 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:52:09.008 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:09.008 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:09.008 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:52:09.009 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:09.009 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:52:09.009 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:09.009 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:52:09.009 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:52:09.009 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:09.009 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:09.009 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:52:09.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:09.010 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:52:09.010 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:52:09.010 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:52:09.010 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:09.010 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:52:09.011 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:52:14.012 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:52:14.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:52:14.014 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:52:14.053 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:52:14.054 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:52:14.426 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:52:14.426 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:52:14.426 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:52:14.426 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:52:14.427 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:52:14.427 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:52:14.427 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:52:14.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:14.427 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:52:14.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:14.428 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:52:14.428 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:14.428 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:14.428 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:52:14.428 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:14.428 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:52:14.428 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:14.429 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:52:14.429 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:52:14.429 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:14.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:14.429 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:14.429 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:52:14.429 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:14.429 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:14.430 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:52:14.430 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:14.430 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:52:14.430 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:14.430 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:52:14.430 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:52:14.430 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:14.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:14.431 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:14.431 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:52:14.431 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:14.431 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:14.431 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:52:14.431 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:14.432 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:52:14.432 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:14.432 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:52:14.432 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:52:14.432 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:14.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:14.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:14.432 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:52:14.433 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:14.433 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:14.433 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:52:14.433 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:14.433 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:52:14.433 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:14.433 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:52:14.434 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:52:14.434 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:14.434 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:14.434 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:52:14.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:14.434 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:52:14.434 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:52:14.434 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:52:14.435 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:14.435 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:52:14.436 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:52:19.437 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:52:19.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:52:19.438 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:52:19.480 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:52:19.480 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:52:19.747 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:52:19.747 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:52:19.747 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:52:19.747 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:52:19.748 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:52:19.748 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:52:19.748 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:52:19.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:19.748 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:52:19.748 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:19.748 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:52:19.749 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:19.749 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:19.749 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:52:19.749 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:19.749 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:52:19.749 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:19.749 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:52:19.750 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:52:19.750 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:19.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:19.750 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:19.750 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:52:19.750 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:19.750 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:19.751 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:52:19.751 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:19.751 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:52:19.751 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:19.751 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:52:19.751 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:52:19.751 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:19.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:19.752 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:19.752 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:52:19.752 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:19.752 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:19.752 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:52:19.752 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:19.753 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:52:19.753 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:19.753 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:52:19.753 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:52:19.753 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:19.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:19.753 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:19.754 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:52:19.754 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:19.754 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:19.754 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:52:19.754 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:19.754 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:52:19.754 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:19.754 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:52:19.755 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:52:19.755 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:19.755 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:19.755 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:52:19.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:19.755 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:52:19.756 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:52:19.756 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:52:19.756 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:19.756 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:52:19.757 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:52:24.758 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:52:24.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:52:24.760 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:52:24.800 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:52:24.801 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:52:25.159 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:52:25.159 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:52:25.159 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:52:25.160 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:52:25.160 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:52:25.160 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:52:25.160 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:52:25.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:25.160 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:52:25.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:25.161 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:52:25.161 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:25.161 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:25.161 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:52:25.161 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:25.161 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:52:25.162 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:25.162 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:52:25.162 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:52:25.162 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:25.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:25.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:25.163 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:52:25.163 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:25.163 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:25.163 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:52:25.163 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:25.163 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:52:25.163 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:25.164 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:52:25.164 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:52:25.164 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:25.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:25.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:25.164 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:52:25.164 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:25.165 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:25.165 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:52:25.165 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:25.165 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:52:25.165 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:25.165 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:52:25.166 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:52:25.166 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:25.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:25.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:25.166 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:52:25.166 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:25.166 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:25.167 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:52:25.167 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:25.167 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:52:25.167 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:25.167 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:52:25.167 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:52:25.167 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:25.168 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:25.168 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:52:25.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:25.168 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:52:25.168 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:52:25.168 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:52:25.168 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:25.169 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:52:25.170 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:52:30.171 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:52:30.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:52:30.173 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:52:30.214 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:52:30.214 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:52:30.580 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:52:30.581 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:52:30.581 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:52:30.581 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:52:30.581 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:52:30.582 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:52:30.582 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:52:30.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:30.583 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:52:30.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:30.583 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:52:30.584 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:30.584 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:30.584 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:52:30.585 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:30.585 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:52:30.585 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:30.585 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:52:30.586 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:52:30.586 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:30.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:30.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:30.587 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:52:30.587 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:30.587 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:30.588 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:52:30.588 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:30.588 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:52:30.589 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:30.589 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:52:30.589 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:52:30.590 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:30.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:30.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:30.590 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:52:30.591 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:30.591 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:30.591 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:52:30.592 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:30.592 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:52:30.592 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:30.592 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:52:30.593 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:52:30.593 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:30.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:30.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:30.594 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:52:30.594 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:30.595 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:30.595 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:52:30.595 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:30.595 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:52:30.596 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:30.596 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:52:30.596 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:52:30.597 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:30.597 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:30.597 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:52:30.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:30.598 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:52:30.598 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:52:30.598 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:52:30.599 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:30.599 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:52:30.600 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:52:35.602 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:52:35.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:52:35.604 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:52:35.645 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:52:35.646 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:52:36.007 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:52:36.007 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:52:36.008 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:52:36.008 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:52:36.008 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:52:36.008 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:52:36.008 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:52:36.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:36.009 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:52:36.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:36.009 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:52:36.009 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:36.009 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:36.009 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:52:36.009 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:36.009 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:52:36.010 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:36.010 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:52:36.010 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:52:36.010 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:36.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:36.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:36.011 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:52:36.011 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:36.011 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:36.011 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:52:36.011 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:36.011 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:52:36.011 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:36.011 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:52:36.012 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:52:36.012 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:36.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:36.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:36.012 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:52:36.012 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:36.012 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:36.013 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:52:36.013 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:36.013 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:52:36.013 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:36.013 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:52:36.013 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:52:36.013 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:36.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:36.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:36.014 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:52:36.014 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:36.014 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:36.014 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:52:36.014 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:36.014 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:52:36.015 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:36.015 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:52:36.015 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:52:36.015 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:36.015 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:36.015 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:52:36.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:36.016 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:52:36.016 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:52:36.016 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:52:36.016 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:36.016 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:52:36.017 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:52:41.019 INFO:tasks.rgw_multisite_tests:test s3 notification set/get/delete on master ... FAIL 2020-11-09T18:52:41.019 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_ps_s3_notification_filter_on_master ... ok 2020-11-09T18:52:41.020 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_ps_s3_notification_filter ... ok 2020-11-09T18:52:41.021 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:52:41.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:52:41.022 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:52:41.061 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:52:41.062 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:52:41.297 INFO:teuthology.orchestra.run.smithi051.stderr:No --period given, using current period=f6258aba-300a-4d5d-ad93-a7c198a0e0a1 2020-11-09T18:52:41.330 INFO:teuthology.orchestra.run.smithi051.stdout:[ 2020-11-09T18:52:41.330 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:52:41.330 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604946340.774139_451.1", 2020-11-09T18:52:41.331 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "2020-11-09T18:25:40.774139Z" 2020-11-09T18:52:41.331 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:52:41.331 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:52:41.331 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:52:41.331 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:52:41.331 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:52:41.331 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:52:41.332 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:52:41.332 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:52:41.332 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:52:41.332 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:52:41.332 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:52:41.332 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:52:41.333 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:52:41.333 INFO:teuthology.orchestra.run.smithi051.stdout:] 2020-11-09T18:52:41.334 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-09T18:52:41.335 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:52:41.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:52:41.336 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:52:41.375 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:52:41.375 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:52:41.637 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:52:41.637 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:52:41.637 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:52:41.638 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:52:41.638 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:52:41.638 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:52:41.638 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:52:41.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:41.638 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:52:41.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:41.639 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:52:41.639 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:41.639 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:41.639 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946340.774139_451.1", 2020-11-09T18:52:41.639 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:41.639 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:52:41.639 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:41.640 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:40.774139Z", 2020-11-09T18:52:41.640 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:52:41.640 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:41.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:41.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:41.640 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:52:41.640 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:41.641 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:41.641 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:52:41.642 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:41.642 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:52:41.642 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:41.642 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:52:41.642 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:52:41.643 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:41.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:41.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:41.643 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:52:41.643 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:41.643 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:41.643 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:52:41.644 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:41.644 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:52:41.644 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:41.644 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:52:41.644 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:52:41.644 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:41.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:41.645 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:41.645 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:52:41.645 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:41.645 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:41.645 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:52:41.646 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:41.646 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:52:41.646 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:41.646 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:52:41.646 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:52:41.646 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:41.646 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:41.647 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:52:41.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:41.647 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:52:41.647 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:52:41.647 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:52:41.647 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:41.647 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:52:41.648 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-09T18:52:41.649 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-09T18:52:41.649 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:52:41.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:52:41.650 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:52:41.690 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:52:41.690 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:52:41.941 INFO:teuthology.orchestra.run.smithi051.stdout:{ 2020-11-09T18:52:41.941 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_status": { 2020-11-09T18:52:41.942 INFO:teuthology.orchestra.run.smithi051.stdout: "info": { 2020-11-09T18:52:41.942 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "sync", 2020-11-09T18:52:41.942 INFO:teuthology.orchestra.run.smithi051.stdout: "num_shards": 4, 2020-11-09T18:52:41.942 INFO:teuthology.orchestra.run.smithi051.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:52:41.942 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 4 2020-11-09T18:52:41.942 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:52:41.943 INFO:teuthology.orchestra.run.smithi051.stdout: "markers": [ 2020-11-09T18:52:41.943 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:52:41.943 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 0, 2020-11-09T18:52:41.943 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:52:41.943 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:52:41.943 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604946340.774139_451.1", 2020-11-09T18:52:41.943 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:52:41.943 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 1, 2020-11-09T18:52:41.944 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:52:41.944 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:25:40.774139Z", 2020-11-09T18:52:41.944 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 4 2020-11-09T18:52:41.944 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:52:41.944 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:52:41.944 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:52:41.945 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 1, 2020-11-09T18:52:41.945 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:52:41.945 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:52:41.945 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:52:41.945 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:52:41.945 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:52:41.945 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:52:41.946 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:52:41.946 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:52:41.946 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:52:41.946 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:52:41.946 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:52:41.946 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 2, 2020-11-09T18:52:41.947 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:52:41.947 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:52:41.947 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:52:41.947 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:52:41.947 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:52:41.947 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:52:41.947 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:52:41.948 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:52:41.948 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:52:41.948 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:52:41.948 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:52:41.948 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 3, 2020-11-09T18:52:41.948 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:52:41.948 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:52:41.949 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:52:41.949 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:52:41.949 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:52:41.949 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:52:41.949 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:52:41.949 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:52:41.950 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:52:41.950 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:52:41.950 INFO:teuthology.orchestra.run.smithi051.stdout: ] 2020-11-09T18:52:41.950 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:52:41.950 INFO:teuthology.orchestra.run.smithi051.stdout: "full_sync": { 2020-11-09T18:52:41.951 INFO:teuthology.orchestra.run.smithi051.stdout: "total": 1, 2020-11-09T18:52:41.951 INFO:teuthology.orchestra.run.smithi051.stdout: "complete": 1 2020-11-09T18:52:41.951 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:52:41.951 INFO:teuthology.orchestra.run.smithi051.stdout:} 2020-11-09T18:52:41.952 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-09T18:52:41.953 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-09T18:52:41.953 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:52:41.953 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:52:41.954 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:52:41.996 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:52:41.996 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:52:42.361 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:52:42.362 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:52:42.362 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:52:42.362 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:52:42.362 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:52:42.362 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:52:42.362 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:52:42.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:42.363 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:52:42.363 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:42.363 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:52:42.363 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:42.363 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:42.363 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:52:42.364 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:42.364 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:52:42.364 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:42.364 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:52:42.365 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:52:42.365 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:42.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:42.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:42.365 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:52:42.365 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:42.365 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:42.366 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:52:42.366 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:42.366 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:52:42.366 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:42.366 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:52:42.366 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:52:42.366 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:42.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:42.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:42.367 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:52:42.367 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:42.367 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:42.367 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:52:42.367 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:42.368 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:52:42.368 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:42.368 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:52:42.368 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:52:42.368 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:42.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:42.368 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:42.369 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:52:42.369 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:42.369 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:42.369 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:52:42.369 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:42.369 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:52:42.369 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:42.369 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:52:42.370 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:52:42.370 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:42.370 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:42.370 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:52:42.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:42.370 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:52:42.370 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:52:42.371 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:52:42.371 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:42.371 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:52:42.372 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:52:47.373 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:52:47.374 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:52:47.374 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:52:47.416 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:52:47.416 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:52:47.786 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:52:47.787 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:52:47.787 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:52:47.787 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:52:47.788 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:52:47.788 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:52:47.788 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:52:47.788 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:47.789 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:52:47.789 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:47.789 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:52:47.792 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:47.792 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:47.793 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:52:47.793 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:47.793 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:52:47.793 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:47.794 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:52:47.794 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:52:47.794 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:47.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:47.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:47.795 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:52:47.796 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:47.796 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:47.796 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:52:47.796 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:47.797 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:52:47.797 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:47.797 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:52:47.798 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:52:47.798 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:47.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:47.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:47.799 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:52:47.799 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:47.799 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:47.800 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:52:47.800 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:47.800 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:52:47.801 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:47.801 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:52:47.801 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:52:47.803 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:47.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:47.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:47.803 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:52:47.803 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:47.803 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:47.804 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:52:47.804 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:47.804 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:52:47.804 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:47.804 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:52:47.804 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:52:47.804 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:47.805 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:47.805 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:52:47.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:47.805 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:52:47.805 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:52:47.805 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:52:47.806 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:47.806 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:52:47.807 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:52:52.808 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:52:52.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:52:52.809 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:52:52.852 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:52:52.852 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:52:53.215 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:52:53.216 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:52:53.216 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:52:53.216 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:52:53.216 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:52:53.216 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:52:53.216 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:52:53.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:53.217 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:52:53.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:53.217 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:52:53.217 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:53.217 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:53.217 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:52:53.217 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:53.218 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:52:53.218 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:53.218 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:52:53.218 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:52:53.218 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:53.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:53.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:53.219 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:52:53.219 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:53.219 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:53.219 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:52:53.219 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:53.219 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:52:53.219 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:53.220 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:52:53.220 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:52:53.220 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:53.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:53.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:53.220 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:52:53.220 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:53.220 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:53.221 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:52:53.221 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:53.221 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:52:53.221 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:53.221 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:52:53.221 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:52:53.221 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:53.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:53.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:53.222 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:52:53.222 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:53.222 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:53.222 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:52:53.223 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:53.223 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:52:53.223 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:53.223 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:52:53.223 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:52:53.223 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:53.223 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:53.224 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:52:53.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:53.224 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:52:53.224 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:52:53.224 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:52:53.224 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:53.224 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:52:53.225 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:52:58.227 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:52:58.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:52:58.228 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:52:58.271 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:52:58.272 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:52:58.634 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:52:58.635 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:52:58.635 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:52:58.635 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:52:58.635 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:52:58.635 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:52:58.635 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:52:58.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:58.636 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:52:58.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:58.636 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:52:58.636 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:58.636 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:58.636 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:52:58.637 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:58.637 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:52:58.637 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:58.637 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:52:58.637 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:52:58.637 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:58.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:58.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:58.638 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:52:58.638 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:58.638 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:58.638 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:52:58.638 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:58.638 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:52:58.639 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:58.639 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:52:58.639 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:52:58.639 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:58.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:58.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:58.639 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:52:58.640 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:58.640 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:58.640 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:52:58.640 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:58.640 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:52:58.640 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:58.640 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:52:58.640 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:52:58.641 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:58.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:58.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:52:58.641 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:52:58.641 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:52:58.641 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:52:58.641 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:52:58.642 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:52:58.642 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:52:58.642 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:52:58.642 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:52:58.642 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:52:58.642 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:58.642 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:58.643 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:52:58.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:52:58.643 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:52:58.643 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:52:58.643 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:52:58.643 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:52:58.643 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:52:58.644 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:53:03.646 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:53:03.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:53:03.647 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:53:03.683 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:53:03.684 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:53:04.044 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:53:04.045 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:53:04.045 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:53:04.045 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:53:04.045 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:53:04.045 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:53:04.045 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:53:04.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:04.046 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:53:04.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:53:04.046 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:53:04.046 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:53:04.047 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:53:04.047 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:53:04.047 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:53:04.047 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:53:04.047 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:53:04.047 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:53:04.047 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:53:04.048 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:04.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:04.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:53:04.048 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:53:04.048 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:53:04.048 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:53:04.049 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:53:04.049 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:53:04.049 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:53:04.049 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:53:04.049 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:53:04.050 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:53:04.050 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:04.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:04.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:53:04.050 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:53:04.050 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:53:04.051 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:53:04.051 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:53:04.051 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:53:04.051 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:53:04.051 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:53:04.051 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:53:04.051 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:53:04.052 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:04.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:04.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:53:04.052 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:53:04.052 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:53:04.052 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:53:04.053 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:53:04.053 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:53:04.053 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:53:04.053 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:53:04.053 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:53:04.053 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:53:04.054 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:04.054 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:04.054 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:53:04.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:04.054 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:53:04.054 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:53:04.055 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:53:04.055 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:04.055 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:53:04.056 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:53:09.057 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:53:09.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:53:09.058 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:53:09.098 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:53:09.099 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:53:09.469 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:53:09.469 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:53:09.470 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:53:09.470 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:53:09.470 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:53:09.470 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:53:09.470 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:53:09.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:09.471 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:53:09.471 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:53:09.471 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:53:09.471 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:53:09.471 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:53:09.471 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:53:09.471 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:53:09.471 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:53:09.472 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:53:09.472 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:53:09.472 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:53:09.472 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:09.472 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:09.472 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:53:09.472 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:53:09.473 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:53:09.473 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:53:09.473 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:53:09.473 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:53:09.473 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:53:09.473 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:53:09.473 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:53:09.474 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:53:09.474 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:09.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:09.474 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:53:09.474 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:53:09.474 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:53:09.474 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:53:09.475 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:53:09.475 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:53:09.475 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:53:09.475 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:53:09.475 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:53:09.475 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:53:09.475 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:09.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:09.476 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:53:09.476 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:53:09.476 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:53:09.476 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:53:09.476 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:53:09.476 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:53:09.476 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:53:09.477 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:53:09.477 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:53:09.477 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:53:09.477 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:09.477 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:09.477 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:53:09.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:09.478 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:53:09.478 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:53:09.478 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:53:09.478 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:09.478 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:53:09.479 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:53:14.481 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:53:14.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:53:14.482 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:53:14.525 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:53:14.525 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:53:14.887 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:53:14.887 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:53:14.887 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:53:14.887 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:53:14.887 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:53:14.888 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:53:14.888 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:53:14.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:14.888 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:53:14.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:53:14.888 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:53:14.888 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:53:14.889 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:53:14.889 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:53:14.889 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:53:14.889 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:53:14.889 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:53:14.889 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:53:14.889 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:53:14.890 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:14.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:14.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:53:14.890 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:53:14.890 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:53:14.890 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:53:14.891 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:53:14.891 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:53:14.891 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:53:14.891 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:53:14.891 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:53:14.891 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:53:14.891 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:14.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:14.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:53:14.892 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:53:14.892 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:53:14.892 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:53:14.892 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:53:14.892 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:53:14.893 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:53:14.893 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:53:14.893 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:53:14.893 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:53:14.893 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:14.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:14.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:53:14.894 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:53:14.894 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:53:14.894 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:53:14.894 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:53:14.895 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:53:14.895 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:53:14.895 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:53:14.895 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:53:14.895 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:53:14.895 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:14.895 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:14.895 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:53:14.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:14.896 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:53:14.896 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:53:14.896 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:53:14.896 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:14.896 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:53:14.897 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:53:19.899 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:53:19.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:53:19.900 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:53:19.942 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:53:19.943 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:53:20.310 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:53:20.311 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:53:20.311 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:53:20.311 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:53:20.311 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:53:20.311 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:53:20.311 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:53:20.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:20.312 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:53:20.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:53:20.312 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:53:20.312 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:53:20.312 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:53:20.312 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:53:20.313 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:53:20.313 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:53:20.313 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:53:20.313 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:53:20.313 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:53:20.313 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:20.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:20.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:53:20.314 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:53:20.314 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:53:20.314 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:53:20.314 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:53:20.314 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:53:20.314 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:53:20.315 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:53:20.315 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:53:20.315 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:53:20.315 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:20.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:20.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:53:20.315 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:53:20.315 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:53:20.316 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:53:20.316 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:53:20.316 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:53:20.316 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:53:20.316 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:53:20.316 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:53:20.316 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:53:20.317 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:20.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:20.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:53:20.317 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:53:20.317 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:53:20.317 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:53:20.317 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:53:20.318 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:53:20.318 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:53:20.318 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:53:20.318 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:53:20.318 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:53:20.318 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:20.318 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:20.319 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:53:20.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:20.319 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:53:20.319 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:53:20.319 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:53:20.319 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:20.319 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:53:20.320 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:53:25.321 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:53:25.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:53:25.323 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:53:25.364 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:53:25.364 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:53:25.731 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:53:25.732 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:53:25.732 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:53:25.732 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:53:25.732 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:53:25.733 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:53:25.733 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:53:25.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:25.733 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:53:25.733 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:53:25.733 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:53:25.733 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:53:25.734 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:53:25.734 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:53:25.735 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:53:25.735 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:53:25.735 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:53:25.735 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:53:25.735 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:53:25.735 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:25.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:25.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:53:25.736 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:53:25.736 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:53:25.736 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:53:25.737 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:53:25.737 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:53:25.737 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:53:25.737 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:53:25.738 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:53:25.738 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:53:25.738 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:25.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:25.739 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:53:25.739 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:53:25.739 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:53:25.739 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:53:25.739 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:53:25.740 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:53:25.740 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:53:25.740 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:53:25.740 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:53:25.740 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:53:25.740 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:25.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:25.741 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:53:25.741 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:53:25.741 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:53:25.741 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:53:25.741 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:53:25.741 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:53:25.742 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:53:25.742 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:53:25.742 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:53:25.743 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:53:25.743 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:25.743 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:25.743 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:53:25.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:25.745 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:53:25.745 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:53:25.745 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:53:25.746 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:25.746 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:53:25.747 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:53:30.749 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:53:30.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:53:30.751 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:53:30.792 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:53:30.792 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:53:31.164 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:53:31.165 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:53:31.165 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:53:31.165 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:53:31.165 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:53:31.165 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:53:31.166 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:53:31.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:31.166 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:53:31.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:53:31.166 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:53:31.167 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:53:31.167 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:53:31.167 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:53:31.167 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:53:31.167 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:53:31.167 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:53:31.167 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:53:31.168 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:53:31.168 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:31.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:31.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:53:31.168 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:53:31.168 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:53:31.169 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:53:31.169 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:53:31.169 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:53:31.169 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:53:31.169 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:53:31.169 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:53:31.169 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:53:31.170 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:31.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:31.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:53:31.170 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:53:31.170 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:53:31.170 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:53:31.170 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:53:31.171 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:53:31.171 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:53:31.171 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:53:31.171 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:53:31.171 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:53:31.171 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:31.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:31.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:53:31.172 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:53:31.172 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:53:31.172 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:53:31.172 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:53:31.172 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:53:31.172 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:53:31.172 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:53:31.173 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:53:31.173 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:53:31.173 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:31.173 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:31.173 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:53:31.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:31.173 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:53:31.174 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:53:31.174 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:53:31.174 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:31.174 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:53:31.175 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:53:36.176 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:53:36.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:53:36.178 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:53:36.219 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:53:36.219 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:53:36.588 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:53:36.588 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:53:36.589 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:53:36.589 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:53:36.589 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:53:36.589 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:53:36.590 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:53:36.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:36.590 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:53:36.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:53:36.591 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:53:36.591 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:53:36.591 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:53:36.591 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:53:36.592 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:53:36.592 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:53:36.592 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:53:36.592 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:53:36.592 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:53:36.593 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:36.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:36.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:53:36.593 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:53:36.593 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:53:36.594 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:53:36.594 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:53:36.594 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:53:36.594 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:53:36.594 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:53:36.595 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:53:36.595 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:53:36.595 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:36.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:36.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:53:36.596 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:53:36.596 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:53:36.596 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:53:36.596 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:53:36.596 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:53:36.597 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:53:36.597 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:53:36.597 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:53:36.597 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:53:36.598 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:36.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:36.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:53:36.598 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:53:36.599 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:53:36.599 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:53:36.599 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:53:36.599 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:53:36.600 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:53:36.600 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:53:36.600 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:53:36.600 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:53:36.601 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:36.601 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:36.601 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:53:36.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:36.602 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:53:36.602 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:53:36.602 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:53:36.602 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:36.603 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:53:36.604 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:53:41.606 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:53:41.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:53:41.608 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:53:41.649 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:53:41.650 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:53:42.006 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:53:42.006 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:53:42.007 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:53:42.007 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:53:42.007 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:53:42.007 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:53:42.007 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:53:42.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:42.008 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:53:42.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:53:42.008 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:53:42.009 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:53:42.009 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:53:42.009 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:53:42.009 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:53:42.009 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:53:42.009 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:53:42.010 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:53:42.010 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:53:42.010 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:42.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:42.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:53:42.011 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:53:42.011 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:53:42.011 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:53:42.011 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:53:42.011 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:53:42.011 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:53:42.012 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:53:42.012 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:53:42.012 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:53:42.012 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:42.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:42.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:53:42.013 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:53:42.013 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:53:42.013 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:53:42.013 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:53:42.013 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:53:42.013 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:53:42.014 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:53:42.014 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:53:42.014 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:53:42.014 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:42.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:42.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:53:42.015 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:53:42.015 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:53:42.015 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:53:42.015 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:53:42.016 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:53:42.017 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:53:42.017 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:53:42.018 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:53:42.018 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:53:42.018 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:42.018 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:42.019 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:53:42.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:42.019 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:53:42.019 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:53:42.019 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:53:42.020 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:42.020 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:53:42.021 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:53:47.022 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:53:47.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:53:47.023 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:53:47.063 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:53:47.063 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:53:47.430 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:53:47.431 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:53:47.431 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:53:47.431 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:53:47.431 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:53:47.431 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:53:47.431 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:53:47.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:47.432 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:53:47.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:53:47.432 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:53:47.432 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:53:47.432 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:53:47.432 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:53:47.433 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:53:47.433 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:53:47.434 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:53:47.434 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:53:47.435 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:53:47.435 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:47.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:47.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:53:47.435 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:53:47.435 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:53:47.435 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:53:47.436 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:53:47.436 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:53:47.436 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:53:47.436 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:53:47.436 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:53:47.436 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:53:47.436 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:47.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:47.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:53:47.437 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:53:47.437 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:53:47.437 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:53:47.437 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:53:47.437 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:53:47.438 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:53:47.438 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:53:47.438 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:53:47.438 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:53:47.438 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:47.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:47.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:53:47.439 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:53:47.439 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:53:47.439 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:53:47.439 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:53:47.439 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:53:47.439 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:53:47.439 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:53:47.440 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:53:47.440 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:53:47.440 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:47.441 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:47.442 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:53:47.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:47.442 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:53:47.442 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:53:47.443 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:53:47.443 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:47.443 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:53:47.444 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:53:52.445 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:53:52.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:53:52.446 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:53:52.488 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:53:52.488 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:53:52.849 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:53:52.850 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:53:52.851 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:53:52.851 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:53:52.851 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:53:52.851 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:53:52.852 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:53:52.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:52.852 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:53:52.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:53:52.853 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:53:52.853 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:53:52.854 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:53:52.854 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:53:52.855 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:53:52.855 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:53:52.855 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:53:52.856 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:53:52.856 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:53:52.856 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:52.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:52.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:53:52.857 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:53:52.858 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:53:52.858 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:53:52.858 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:53:52.859 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:53:52.859 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:53:52.859 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:53:52.860 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:53:52.860 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:53:52.861 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:52.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:52.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:53:52.862 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:53:52.862 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:53:52.862 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:53:52.863 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:53:52.863 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:53:52.863 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:53:52.864 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:53:52.864 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:53:52.864 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:53:52.865 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:52.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:52.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:53:52.866 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:53:52.866 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:53:52.866 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:53:52.867 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:53:52.867 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:53:52.867 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:53:52.867 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:53:52.868 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:53:52.868 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:53:52.868 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:52.869 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:52.869 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:53:52.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:52.870 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:53:52.870 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:53:52.870 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:53:52.871 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:52.871 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:53:52.873 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:53:57.874 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:53:57.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:53:57.876 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:53:57.919 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:53:57.919 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:53:58.293 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:53:58.293 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:53:58.294 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:53:58.294 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:53:58.294 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:53:58.294 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:53:58.294 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:53:58.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:58.294 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:53:58.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:53:58.295 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:53:58.295 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:53:58.295 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:53:58.295 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:53:58.295 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:53:58.295 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:53:58.296 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:53:58.296 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:53:58.296 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:53:58.296 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:58.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:58.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:53:58.296 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:53:58.297 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:53:58.297 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:53:58.297 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:53:58.297 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:53:58.297 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:53:58.297 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:53:58.298 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:53:58.298 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:53:58.298 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:58.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:58.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:53:58.298 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:53:58.298 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:53:58.299 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:53:58.299 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:53:58.299 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:53:58.299 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:53:58.299 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:53:58.299 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:53:58.299 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:53:58.299 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:58.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:58.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:53:58.300 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:53:58.300 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:53:58.300 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:53:58.300 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:53:58.300 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:53:58.301 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:53:58.301 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:53:58.301 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:53:58.301 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:53:58.301 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:58.301 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:58.301 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:53:58.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:53:58.302 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:53:58.302 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:53:58.302 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:53:58.302 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:53:58.302 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:53:58.303 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:54:03.304 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:54:03.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:54:03.305 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:54:03.346 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:54:03.347 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:54:03.710 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:54:03.710 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:54:03.711 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:54:03.711 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:54:03.711 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:54:03.711 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:54:03.711 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:54:03.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:03.711 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:54:03.712 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:54:03.712 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:54:03.712 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:54:03.712 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:54:03.712 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:54:03.712 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:54:03.712 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:54:03.713 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:54:03.713 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:54:03.713 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:54:03.713 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:03.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:03.713 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:54:03.713 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:54:03.714 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:54:03.714 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:54:03.714 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:54:03.714 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:54:03.715 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:54:03.715 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:54:03.715 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:54:03.715 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:54:03.715 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:03.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:03.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:54:03.716 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:54:03.716 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:54:03.716 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:54:03.716 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:54:03.716 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:54:03.716 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:54:03.716 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:54:03.717 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:54:03.717 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:54:03.717 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:03.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:03.717 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:54:03.717 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:54:03.717 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:54:03.717 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:54:03.718 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:54:03.719 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:54:03.719 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:54:03.719 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:54:03.719 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:54:03.719 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:54:03.719 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:03.720 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:03.720 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:54:03.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:03.720 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:54:03.720 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:54:03.720 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:54:03.720 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:03.721 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:54:03.722 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:54:08.725 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:54:08.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:54:08.726 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:54:08.766 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:54:08.766 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:54:09.129 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:54:09.129 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:54:09.130 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:54:09.130 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:54:09.130 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:54:09.130 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:54:09.130 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:54:09.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:09.131 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:54:09.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:54:09.131 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:54:09.131 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:54:09.131 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:54:09.131 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:54:09.131 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:54:09.132 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:54:09.132 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:54:09.132 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:54:09.132 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:54:09.132 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:09.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:09.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:54:09.133 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:54:09.133 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:54:09.133 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:54:09.133 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:54:09.133 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:54:09.133 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:54:09.133 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:54:09.134 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:54:09.134 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:54:09.134 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:09.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:09.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:54:09.135 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:54:09.135 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:54:09.135 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:54:09.135 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:54:09.136 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:54:09.136 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:54:09.136 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:54:09.136 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:54:09.136 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:54:09.136 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:09.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:09.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:54:09.138 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:54:09.138 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:54:09.138 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:54:09.138 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:54:09.138 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:54:09.138 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:54:09.139 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:54:09.139 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:54:09.139 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:54:09.139 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:09.139 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:09.139 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:54:09.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:09.140 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:54:09.140 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:54:09.140 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:54:09.141 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:09.141 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:54:09.142 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:54:14.143 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:54:14.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:54:14.144 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:54:14.183 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:54:14.184 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:54:14.550 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:54:14.551 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:54:14.551 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:54:14.551 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:54:14.551 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:54:14.551 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:54:14.552 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:54:14.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:14.552 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:54:14.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:54:14.552 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:54:14.553 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:54:14.553 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:54:14.553 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:54:14.553 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:54:14.553 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:54:14.553 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:54:14.554 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:54:14.554 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:54:14.554 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:14.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:14.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:54:14.555 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:54:14.556 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:54:14.556 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:54:14.556 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:54:14.556 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:54:14.556 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:54:14.556 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:54:14.557 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:54:14.557 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:54:14.557 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:14.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:14.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:54:14.557 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:54:14.557 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:54:14.558 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:54:14.558 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:54:14.558 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:54:14.558 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:54:14.558 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:54:14.558 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:54:14.559 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:54:14.559 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:14.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:14.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:54:14.559 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:54:14.559 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:54:14.559 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:54:14.560 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:54:14.560 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:54:14.560 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:54:14.560 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:54:14.560 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:54:14.560 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:54:14.560 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:14.561 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:14.561 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:54:14.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:14.561 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:54:14.561 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:54:14.561 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:54:14.562 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:14.562 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:54:14.563 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:54:19.567 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:54:19.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:54:19.569 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:54:19.617 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:54:19.617 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:54:19.967 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:54:19.969 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:54:19.969 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:54:19.969 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:54:19.969 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:54:19.970 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:54:19.970 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:54:19.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:19.970 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:54:19.970 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:54:19.970 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:54:19.970 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:54:19.971 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:54:19.971 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:54:19.971 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:54:19.971 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:54:19.971 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:54:19.971 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:54:19.971 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:54:19.972 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:19.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:19.972 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:54:19.972 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:54:19.972 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:54:19.972 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:54:19.972 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:54:19.973 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:54:19.973 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:54:19.973 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:54:19.973 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:54:19.973 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:54:19.973 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:19.973 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:19.974 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:54:19.974 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:54:19.974 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:54:19.974 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:54:19.974 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:54:19.974 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:54:19.974 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:54:19.974 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:54:19.975 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:54:19.975 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:54:19.975 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:19.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:19.975 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:54:19.975 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:54:19.975 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:54:19.976 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:54:19.976 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:54:19.976 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:54:19.976 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:54:19.976 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:54:19.976 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:54:19.976 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:54:19.977 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:19.977 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:19.977 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:54:19.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:19.977 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:54:19.977 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:54:19.978 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:54:19.978 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:19.978 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:54:19.979 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:54:24.980 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:54:24.981 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:54:24.981 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:54:25.022 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:54:25.022 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:54:25.383 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:54:25.384 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:54:25.384 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:54:25.384 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:54:25.384 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:54:25.384 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:54:25.385 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:54:25.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:25.385 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:54:25.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:54:25.385 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:54:25.385 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:54:25.386 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:54:25.386 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:54:25.386 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:54:25.386 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:54:25.386 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:54:25.386 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:54:25.386 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:54:25.387 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:25.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:25.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:54:25.387 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:54:25.387 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:54:25.387 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:54:25.387 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:54:25.388 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:54:25.388 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:54:25.388 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:54:25.388 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:54:25.388 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:54:25.388 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:25.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:25.388 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:54:25.389 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:54:25.389 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:54:25.389 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:54:25.389 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:54:25.389 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:54:25.389 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:54:25.389 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:54:25.390 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:54:25.390 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:54:25.390 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:25.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:25.390 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:54:25.390 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:54:25.390 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:54:25.391 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:54:25.391 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:54:25.391 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:54:25.391 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:54:25.391 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:54:25.391 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:54:25.391 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:54:25.391 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:25.392 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:25.392 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:54:25.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:25.392 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:54:25.392 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:54:25.392 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:54:25.392 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:25.393 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:54:25.393 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:54:30.395 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:54:30.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:54:30.396 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:54:30.436 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:54:30.436 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:54:30.803 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:54:30.803 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:54:30.804 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:54:30.804 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:54:30.804 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:54:30.804 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:54:30.804 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:54:30.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:30.805 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:54:30.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:54:30.805 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:54:30.805 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:54:30.806 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:54:30.806 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:54:30.806 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:54:30.806 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:54:30.806 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:54:30.806 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:54:30.807 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:54:30.807 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:30.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:30.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:54:30.807 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:54:30.807 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:54:30.807 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:54:30.808 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:54:30.808 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:54:30.808 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:54:30.808 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:54:30.808 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:54:30.808 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:54:30.809 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:30.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:30.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:54:30.809 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:54:30.809 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:54:30.809 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:54:30.810 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:54:30.810 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:54:30.810 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:54:30.810 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:54:30.810 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:54:30.810 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:54:30.810 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:30.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:30.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:54:30.811 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:54:30.811 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:54:30.811 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:54:30.811 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:54:30.812 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:54:30.812 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:54:30.812 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:54:30.812 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:54:30.812 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:54:30.812 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:30.812 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:30.813 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:54:30.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:30.813 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:54:30.813 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:54:30.813 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:54:30.813 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:30.813 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:54:30.814 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:54:35.816 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:54:35.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:54:35.818 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:54:35.858 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:54:35.859 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:54:36.232 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:54:36.232 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:54:36.232 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:54:36.232 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:54:36.232 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:54:36.233 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:54:36.233 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:54:36.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:36.233 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:54:36.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:54:36.233 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:54:36.233 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:54:36.234 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:54:36.234 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:54:36.234 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:54:36.234 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:54:36.234 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:54:36.234 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:54:36.234 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:54:36.235 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:36.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:36.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:54:36.235 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:54:36.235 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:54:36.235 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:54:36.235 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:54:36.236 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:54:36.236 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:54:36.236 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:54:36.236 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:54:36.236 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:54:36.236 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:36.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:36.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:54:36.237 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:54:36.237 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:54:36.237 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:54:36.237 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:54:36.237 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:54:36.237 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:54:36.237 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:54:36.238 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:54:36.238 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:54:36.238 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:36.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:36.238 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:54:36.238 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:54:36.238 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:54:36.239 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:54:36.239 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:54:36.239 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:54:36.239 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:54:36.239 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:54:36.239 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:54:36.239 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:54:36.240 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:36.240 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:36.240 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:54:36.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:36.240 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:54:36.240 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:54:36.240 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:54:36.241 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:36.241 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:54:36.242 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:54:41.244 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:54:41.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:54:41.245 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:54:41.286 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:54:41.287 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:54:41.647 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:54:41.696 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:54:41.696 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:54:41.696 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:54:41.696 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:54:41.697 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:54:41.697 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:54:41.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:41.697 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:54:41.697 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:54:41.697 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:54:41.697 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:54:41.697 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:54:41.698 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:54:41.698 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:54:41.698 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:54:41.699 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:54:41.699 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:54:41.699 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:54:41.699 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:41.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:41.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:54:41.700 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:54:41.700 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:54:41.700 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:54:41.700 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:54:41.700 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:54:41.700 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:54:41.701 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:54:41.701 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:54:41.701 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:54:41.701 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:41.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:41.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:54:41.701 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:54:41.702 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:54:41.702 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:54:41.702 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:54:41.702 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:54:41.702 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:54:41.702 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:54:41.703 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:54:41.703 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:54:41.703 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:41.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:41.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:54:41.703 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:54:41.703 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:54:41.704 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:54:41.704 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:54:41.704 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:54:41.704 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:54:41.704 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:54:41.704 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:54:41.704 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:54:41.704 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:41.705 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:41.705 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:54:41.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:41.705 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:54:41.705 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:54:41.705 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:54:41.705 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:41.706 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:54:41.707 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:54:46.708 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:54:46.709 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:54:46.710 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:54:46.756 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:54:46.756 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:54:47.224 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:54:47.224 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:54:47.224 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:54:47.224 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:54:47.225 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:54:47.225 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:54:47.225 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:54:47.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:47.225 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:54:47.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:54:47.226 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:54:47.226 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:54:47.226 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:54:47.226 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:54:47.227 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:54:47.227 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:54:47.227 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:54:47.227 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:54:47.228 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:54:47.228 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:47.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:47.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:54:47.229 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:54:47.229 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:54:47.229 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:54:47.230 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:54:47.230 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:54:47.230 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:54:47.230 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:54:47.231 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:54:47.231 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:54:47.231 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:47.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:47.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:54:47.232 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:54:47.232 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:54:47.233 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:54:47.233 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:54:47.233 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:54:47.233 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:54:47.234 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:54:47.234 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:54:47.234 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:54:47.234 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:47.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:47.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:54:47.235 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:54:47.236 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:54:47.236 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:54:47.236 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:54:47.236 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:54:47.237 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:54:47.237 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:54:47.237 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:54:47.238 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:54:47.238 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:47.238 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:47.238 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:54:47.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:47.239 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:54:47.239 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:54:47.240 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:54:47.240 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:47.240 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:54:47.242 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:54:52.244 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:54:52.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:54:52.245 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:54:52.286 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:54:52.286 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:54:52.662 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:54:52.662 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:54:52.662 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:54:52.662 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:54:52.663 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:54:52.663 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:54:52.663 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:54:52.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:52.663 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:54:52.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:54:52.664 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:54:52.665 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:54:52.665 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:54:52.665 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:54:52.665 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:54:52.665 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:54:52.665 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:54:52.665 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:54:52.666 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:54:52.666 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:52.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:52.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:54:52.666 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:54:52.666 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:54:52.667 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:54:52.667 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:54:52.667 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:54:52.667 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:54:52.667 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:54:52.667 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:54:52.667 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:54:52.667 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:52.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:52.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:54:52.668 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:54:52.668 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:54:52.668 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:54:52.668 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:54:52.668 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:54:52.669 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:54:52.669 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:54:52.669 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:54:52.669 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:54:52.669 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:52.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:52.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:54:52.670 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:54:52.670 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:54:52.670 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:54:52.670 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:54:52.670 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:54:52.670 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:54:52.670 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:54:52.671 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:54:52.671 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:54:52.671 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:52.671 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:52.671 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:54:52.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:52.671 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:54:52.671 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:54:52.672 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:54:52.672 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:52.672 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:54:52.674 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:54:57.675 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:54:57.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:54:57.677 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:54:57.718 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:54:57.719 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:54:58.087 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:54:58.088 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:54:58.088 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:54:58.088 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:54:58.088 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:54:58.088 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:54:58.089 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:54:58.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:58.089 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:54:58.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:54:58.089 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:54:58.089 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:54:58.089 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:54:58.090 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:54:58.090 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:54:58.090 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:54:58.090 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:54:58.090 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:54:58.090 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:54:58.090 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:58.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:58.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:54:58.091 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:54:58.091 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:54:58.091 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:54:58.091 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:54:58.091 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:54:58.092 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:54:58.092 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:54:58.092 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:54:58.092 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:54:58.092 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:58.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:58.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:54:58.092 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:54:58.093 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:54:58.093 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:54:58.093 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:54:58.093 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:54:58.093 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:54:58.093 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:54:58.093 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:54:58.094 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:54:58.094 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:58.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:58.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:54:58.094 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:54:58.094 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:54:58.094 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:54:58.095 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:54:58.095 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:54:58.095 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:54:58.095 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:54:58.095 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:54:58.095 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:54:58.095 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:58.095 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:58.096 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:54:58.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:54:58.096 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:54:58.096 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:54:58.096 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:54:58.096 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:54:58.096 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:54:58.097 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:55:03.099 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:55:03.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:55:03.100 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:55:03.142 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:55:03.143 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:55:03.514 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:55:03.514 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:55:03.514 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:55:03.515 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:55:03.515 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:55:03.515 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:55:03.515 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:55:03.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:03.515 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:55:03.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:55:03.516 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:55:03.516 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:55:03.516 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:55:03.516 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:55:03.516 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:55:03.516 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:55:03.516 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:55:03.517 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:55:03.517 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:55:03.517 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:03.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:03.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:55:03.517 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:55:03.517 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:55:03.518 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:55:03.518 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:55:03.518 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:55:03.518 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:55:03.518 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:55:03.518 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:55:03.518 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:55:03.518 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:03.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:03.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:55:03.519 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:55:03.519 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:55:03.519 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:55:03.519 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:55:03.519 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:55:03.520 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:55:03.520 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:55:03.520 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:55:03.520 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:55:03.520 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:03.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:03.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:55:03.521 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:55:03.521 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:55:03.521 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:55:03.521 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:55:03.521 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:55:03.521 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:55:03.521 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:55:03.521 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:55:03.522 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:55:03.522 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:03.522 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:03.522 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:55:03.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:03.522 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:55:03.522 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:55:03.523 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:55:03.523 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:03.523 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:55:03.524 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:55:08.525 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:55:08.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:55:08.526 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:55:08.573 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:55:08.573 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:55:09.024 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:55:09.024 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:55:09.024 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:55:09.025 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:55:09.025 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:55:09.025 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:55:09.025 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:55:09.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:09.025 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:55:09.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:55:09.026 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:55:09.026 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:55:09.026 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:55:09.026 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:55:09.026 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:55:09.026 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:55:09.027 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:55:09.027 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:55:09.027 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:55:09.027 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:09.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:09.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:55:09.028 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:55:09.028 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:55:09.028 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:55:09.028 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:55:09.028 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:55:09.028 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:55:09.028 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:55:09.029 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:55:09.029 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:55:09.029 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:09.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:09.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:55:09.029 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:55:09.029 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:55:09.030 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:55:09.030 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:55:09.030 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:55:09.030 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:55:09.030 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:55:09.030 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:55:09.030 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:55:09.031 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:09.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:09.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:55:09.031 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:55:09.031 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:55:09.031 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:55:09.031 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:55:09.032 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:55:09.032 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:55:09.032 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:55:09.032 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:55:09.032 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:55:09.032 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:09.032 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:09.033 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:55:09.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:09.033 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:55:09.033 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:55:09.033 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:55:09.033 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:09.033 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:55:09.034 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:55:14.035 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:55:14.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:55:14.037 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:55:14.078 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:55:14.078 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:55:14.447 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:55:14.448 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:55:14.448 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:55:14.448 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:55:14.448 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:55:14.448 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:55:14.448 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:55:14.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:14.449 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:55:14.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:55:14.449 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:55:14.449 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:55:14.449 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:55:14.449 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:55:14.450 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:55:14.450 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:55:14.450 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:55:14.450 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:55:14.450 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:55:14.450 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:14.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:14.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:55:14.451 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:55:14.451 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:55:14.451 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:55:14.451 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:55:14.451 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:55:14.451 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:55:14.452 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:55:14.452 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:55:14.452 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:55:14.452 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:14.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:14.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:55:14.452 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:55:14.453 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:55:14.453 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:55:14.453 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:55:14.453 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:55:14.453 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:55:14.453 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:55:14.453 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:55:14.454 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:55:14.454 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:14.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:14.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:55:14.454 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:55:14.454 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:55:14.454 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:55:14.455 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:55:14.455 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:55:14.455 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:55:14.455 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:55:14.455 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:55:14.456 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:55:14.456 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:14.456 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:14.456 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:55:14.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:14.456 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:55:14.456 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:55:14.457 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:55:14.457 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:14.458 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:55:14.459 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:55:19.461 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:55:19.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:55:19.462 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:55:19.503 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:55:19.503 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:55:19.862 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:55:19.862 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:55:19.863 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:55:19.863 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:55:19.863 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:55:19.863 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:55:19.863 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:55:19.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:19.864 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:55:19.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:55:19.864 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:55:19.864 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:55:19.865 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:55:19.865 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:55:19.865 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:55:19.865 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:55:19.865 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:55:19.866 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:55:19.866 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:55:19.866 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:19.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:19.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:55:19.867 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:55:19.867 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:55:19.867 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:55:19.867 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:55:19.867 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:55:19.868 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:55:19.868 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:55:19.868 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:55:19.868 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:55:19.868 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:19.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:19.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:55:19.869 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:55:19.869 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:55:19.869 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:55:19.869 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:55:19.869 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:55:19.869 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:55:19.870 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:55:19.870 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:55:19.870 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:55:19.870 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:19.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:19.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:55:19.871 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:55:19.871 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:55:19.871 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:55:19.872 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:55:19.872 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:55:19.872 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:55:19.873 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:55:19.873 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:55:19.873 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:55:19.873 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:19.873 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:19.873 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:55:19.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:19.874 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:55:19.874 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:55:19.874 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:55:19.874 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:19.874 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:55:19.876 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:55:24.876 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:55:24.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:55:24.878 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:55:24.921 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:55:24.921 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:55:25.286 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:55:25.287 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:55:25.287 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:55:25.287 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:55:25.287 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:55:25.287 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:55:25.287 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:55:25.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:25.288 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:55:25.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:55:25.288 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:55:25.288 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:55:25.288 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:55:25.288 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:55:25.288 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:55:25.289 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:55:25.289 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:55:25.289 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:55:25.289 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:55:25.289 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:25.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:25.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:55:25.290 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:55:25.290 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:55:25.290 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:55:25.290 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:55:25.290 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:55:25.290 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:55:25.291 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:55:25.291 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:55:25.291 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:55:25.291 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:25.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:25.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:55:25.291 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:55:25.292 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:55:25.292 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:55:25.292 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:55:25.292 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:55:25.292 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:55:25.292 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:55:25.292 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:55:25.293 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:55:25.293 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:25.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:25.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:55:25.293 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:55:25.293 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:55:25.293 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:55:25.294 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:55:25.294 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:55:25.294 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:55:25.294 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:55:25.294 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:55:25.294 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:55:25.294 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:25.294 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:25.295 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:55:25.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:25.295 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:55:25.295 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:55:25.295 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:55:25.295 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:25.295 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:55:25.296 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:55:30.297 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:55:30.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:55:30.299 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:55:30.340 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:55:30.340 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:55:30.707 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:55:30.708 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:55:30.708 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:55:30.708 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:55:30.708 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:55:30.708 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:55:30.708 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:55:30.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:30.709 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:55:30.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:55:30.709 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:55:30.709 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:55:30.709 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:55:30.709 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:55:30.710 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:55:30.710 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:55:30.710 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:55:30.710 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:55:30.710 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:55:30.710 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:30.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:30.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:55:30.711 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:55:30.711 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:55:30.711 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:55:30.711 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:55:30.711 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:55:30.711 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:55:30.712 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:55:30.712 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:55:30.712 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:55:30.712 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:30.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:30.712 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:55:30.712 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:55:30.713 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:55:30.713 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:55:30.713 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:55:30.713 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:55:30.713 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:55:30.713 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:55:30.713 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:55:30.714 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:55:30.714 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:30.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:30.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:55:30.714 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:55:30.714 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:55:30.714 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:55:30.714 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:55:30.715 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:55:30.715 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:55:30.715 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:55:30.715 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:55:30.715 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:55:30.715 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:30.715 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:30.716 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:55:30.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:30.716 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:55:30.716 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:55:30.716 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:55:30.716 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:30.716 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:55:30.717 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:55:35.719 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:55:35.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:55:35.720 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:55:35.760 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:55:35.761 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:55:36.125 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:55:36.125 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:55:36.125 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:55:36.125 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:55:36.126 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:55:36.126 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:55:36.126 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:55:36.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:36.126 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:55:36.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:55:36.127 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:55:36.127 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:55:36.127 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:55:36.127 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:55:36.127 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:55:36.127 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:55:36.127 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:55:36.128 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:55:36.128 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:55:36.128 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:36.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:36.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:55:36.128 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:55:36.128 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:55:36.128 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:55:36.129 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:55:36.129 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:55:36.129 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:55:36.129 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:55:36.129 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:55:36.129 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:55:36.129 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:36.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:36.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:55:36.130 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:55:36.130 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:55:36.130 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:55:36.130 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:55:36.130 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:55:36.131 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:55:36.131 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:55:36.131 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:55:36.131 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:55:36.131 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:36.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:36.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:55:36.132 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:55:36.132 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:55:36.132 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:55:36.132 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:55:36.132 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:55:36.132 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:55:36.132 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:55:36.132 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:55:36.133 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:55:36.133 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:36.133 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:36.133 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:55:36.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:36.133 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:55:36.133 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:55:36.134 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:55:36.134 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:36.134 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:55:36.135 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:55:41.136 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:55:41.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:55:41.137 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:55:41.179 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:55:41.180 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:55:41.444 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:55:41.444 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:55:41.445 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:55:41.445 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:55:41.445 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:55:41.445 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:55:41.445 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:55:41.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:41.446 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:55:41.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:55:41.446 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:55:41.446 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:55:41.446 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:55:41.446 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:55:41.446 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:55:41.447 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:55:41.447 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:55:41.447 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:55:41.447 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:55:41.447 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:41.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:41.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:55:41.448 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:55:41.448 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:55:41.448 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:55:41.448 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:55:41.448 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:55:41.448 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:55:41.448 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:55:41.449 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:55:41.449 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:55:41.449 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:41.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:41.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:55:41.449 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:55:41.449 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:55:41.449 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:55:41.450 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:55:41.450 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:55:41.450 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:55:41.450 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:55:41.450 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:55:41.450 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:55:41.450 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:41.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:41.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:55:41.451 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:55:41.451 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:55:41.451 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:55:41.451 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:55:41.452 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:55:41.452 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:55:41.452 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:55:41.452 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:55:41.452 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:55:41.452 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:41.452 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:41.453 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:55:41.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:41.453 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:55:41.453 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:55:41.453 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:55:41.453 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:41.453 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:55:41.454 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:55:46.456 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:55:46.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:55:46.457 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:55:46.498 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:55:46.498 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:55:46.864 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:55:46.865 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:55:46.865 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:55:46.865 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:55:46.865 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:55:46.865 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:55:46.866 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:55:46.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:46.866 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:55:46.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:55:46.866 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:55:46.866 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:55:46.867 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:55:46.867 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:55:46.867 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:55:46.867 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:55:46.867 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:55:46.867 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:55:46.867 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:55:46.868 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:46.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:46.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:55:46.868 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:55:46.868 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:55:46.868 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:55:46.868 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:55:46.868 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:55:46.869 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:55:46.869 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:55:46.869 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:55:46.869 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:55:46.869 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:46.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:46.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:55:46.870 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:55:46.870 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:55:46.870 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:55:46.870 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:55:46.870 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:55:46.870 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:55:46.870 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:55:46.871 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:55:46.871 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:55:46.871 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:46.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:46.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:55:46.871 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:55:46.871 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:55:46.872 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:55:46.872 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:55:46.872 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:55:46.872 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:55:46.872 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:55:46.872 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:55:46.872 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:55:46.873 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:46.873 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:46.873 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:55:46.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:46.873 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:55:46.873 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:55:46.873 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:55:46.873 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:46.874 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:55:46.875 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:55:51.876 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:55:51.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:55:51.877 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:55:51.920 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:55:51.921 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:55:52.287 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:55:52.287 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:55:52.288 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:55:52.288 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:55:52.288 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:55:52.288 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:55:52.288 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:55:52.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:52.289 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:55:52.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:55:52.289 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:55:52.289 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:55:52.289 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:55:52.289 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:55:52.290 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:55:52.290 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:55:52.290 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:55:52.290 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:55:52.291 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:55:52.291 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:52.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:52.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:55:52.291 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:55:52.291 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:55:52.291 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:55:52.292 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:55:52.292 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:55:52.292 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:55:52.292 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:55:52.292 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:55:52.292 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:55:52.293 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:52.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:52.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:55:52.293 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:55:52.293 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:55:52.293 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:55:52.293 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:55:52.294 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:55:52.294 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:55:52.294 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:55:52.294 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:55:52.294 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:55:52.294 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:52.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:52.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:55:52.295 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:55:52.295 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:55:52.295 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:55:52.295 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:55:52.296 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:55:52.296 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:55:52.296 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:55:52.296 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:55:52.296 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:55:52.296 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:52.296 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:52.297 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:55:52.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:52.297 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:55:52.297 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:55:52.297 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:55:52.297 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:52.297 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:55:52.298 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:55:57.300 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:55:57.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:55:57.302 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:55:57.343 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:55:57.343 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:55:57.719 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:55:57.720 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:55:57.720 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:55:57.720 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:55:57.721 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:55:57.721 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:55:57.721 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:55:57.722 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:57.722 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:55:57.722 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:55:57.722 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:55:57.723 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:55:57.723 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:55:57.723 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:55:57.724 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:55:57.724 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:55:57.724 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:55:57.724 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:55:57.725 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:55:57.725 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:57.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:57.726 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:55:57.726 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:55:57.727 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:55:57.727 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:55:57.727 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:55:57.727 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:55:57.728 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:55:57.728 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:55:57.728 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:55:57.729 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:55:57.729 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:57.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:57.730 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:55:57.730 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:55:57.730 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:55:57.731 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:55:57.731 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:55:57.731 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:55:57.732 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:55:57.732 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:55:57.733 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:55:57.733 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:55:57.733 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:57.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:57.734 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:55:57.734 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:55:57.735 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:55:57.735 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:55:57.735 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:55:57.735 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:55:57.736 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:55:57.736 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:55:57.736 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:55:57.736 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:55:57.737 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:57.737 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:57.737 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:55:57.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:55:57.738 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:55:57.738 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:55:57.738 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:55:57.739 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:55:57.739 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:55:57.740 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:56:02.742 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:56:02.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:56:02.743 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:56:02.783 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:56:02.784 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:56:03.148 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:56:03.148 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:56:03.148 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:56:03.149 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:56:03.149 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:56:03.149 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:56:03.149 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:56:03.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:03.149 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:56:03.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:56:03.150 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:56:03.150 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:56:03.150 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:56:03.150 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:56:03.150 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:56:03.150 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:56:03.150 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:56:03.151 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:56:03.151 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:56:03.151 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:03.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:03.151 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:56:03.151 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:56:03.151 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:56:03.152 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:56:03.152 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:56:03.152 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:56:03.152 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:56:03.152 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:56:03.152 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:56:03.152 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:56:03.153 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:03.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:03.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:56:03.153 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:56:03.153 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:56:03.153 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:56:03.153 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:56:03.154 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:56:03.154 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:56:03.154 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:56:03.154 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:56:03.154 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:56:03.154 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:03.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:03.155 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:56:03.155 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:56:03.155 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:56:03.155 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:56:03.155 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:56:03.156 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:56:03.156 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:56:03.156 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:56:03.156 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:56:03.156 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:56:03.156 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:03.156 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:03.157 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:56:03.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:03.157 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:56:03.157 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:56:03.157 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:56:03.157 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:03.157 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:56:03.158 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:56:08.159 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:56:08.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:56:08.161 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:56:08.204 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:56:08.204 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:56:08.577 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:56:08.577 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:56:08.578 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:56:08.578 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:56:08.578 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:56:08.579 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:56:08.579 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:56:08.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:08.580 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:56:08.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:56:08.580 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:56:08.581 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:56:08.581 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:56:08.581 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:56:08.582 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:56:08.582 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:56:08.582 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:56:08.584 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:56:08.584 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:56:08.584 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:08.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:08.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:56:08.585 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:56:08.585 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:56:08.586 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:56:08.586 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:56:08.586 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:56:08.587 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:56:08.587 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:56:08.587 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:56:08.587 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:56:08.588 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:08.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:08.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:56:08.588 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:56:08.588 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:56:08.588 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:56:08.589 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:56:08.589 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:56:08.589 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:56:08.589 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:56:08.590 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:56:08.590 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:56:08.590 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:08.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:08.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:56:08.591 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:56:08.591 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:56:08.591 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:56:08.591 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:56:08.592 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:56:08.592 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:56:08.592 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:56:08.592 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:56:08.592 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:56:08.592 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:08.592 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:08.593 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:56:08.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:08.593 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:56:08.593 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:56:08.593 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:56:08.593 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:08.594 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:56:08.595 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:56:13.596 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:56:13.597 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:56:13.597 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:56:13.645 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:56:13.646 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:56:14.002 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:56:14.002 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:56:14.002 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:56:14.002 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:56:14.003 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:56:14.003 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:56:14.003 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:56:14.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:14.003 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:56:14.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:56:14.003 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:56:14.004 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:56:14.004 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:56:14.004 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:56:14.004 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:56:14.004 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:56:14.004 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:56:14.005 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:56:14.005 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:56:14.005 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:14.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:14.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:56:14.005 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:56:14.005 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:56:14.005 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:56:14.006 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:56:14.006 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:56:14.006 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:56:14.006 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:56:14.006 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:56:14.006 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:56:14.007 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:14.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:14.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:56:14.007 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:56:14.007 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:56:14.007 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:56:14.007 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:56:14.007 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:56:14.008 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:56:14.008 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:56:14.008 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:56:14.008 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:56:14.008 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:14.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:14.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:56:14.009 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:56:14.009 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:56:14.009 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:56:14.009 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:56:14.009 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:56:14.009 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:56:14.009 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:56:14.009 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:56:14.010 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:56:14.010 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:14.010 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:14.010 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:56:14.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:14.010 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:56:14.010 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:56:14.011 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:56:14.011 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:14.011 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:56:14.012 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:56:19.013 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:56:19.014 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:56:19.015 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:56:19.055 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:56:19.056 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:56:19.432 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:56:19.433 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:56:19.433 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:56:19.433 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:56:19.433 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:56:19.433 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:56:19.434 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:56:19.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:19.434 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:56:19.434 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:56:19.434 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:56:19.434 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:56:19.434 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:56:19.435 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:56:19.435 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:56:19.435 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:56:19.435 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:56:19.435 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:56:19.435 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:56:19.435 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:19.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:19.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:56:19.436 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:56:19.436 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:56:19.436 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:56:19.436 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:56:19.436 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:56:19.436 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:56:19.437 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:56:19.437 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:56:19.437 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:56:19.437 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:19.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:19.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:56:19.437 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:56:19.438 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:56:19.438 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:56:19.438 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:56:19.438 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:56:19.438 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:56:19.438 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:56:19.438 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:56:19.439 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:56:19.439 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:19.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:19.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:56:19.439 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:56:19.439 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:56:19.439 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:56:19.439 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:56:19.440 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:56:19.440 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:56:19.440 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:56:19.440 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:56:19.440 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:56:19.440 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:19.441 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:19.441 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:56:19.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:19.441 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:56:19.441 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:56:19.442 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:56:19.442 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:19.442 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:56:19.443 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:56:24.444 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:56:24.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:56:24.446 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:56:24.486 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:56:24.487 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:56:24.856 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:56:24.857 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:56:24.857 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:56:24.857 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:56:24.857 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:56:24.857 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:56:24.857 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:56:24.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:24.858 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:56:24.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:56:24.858 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:56:24.858 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:56:24.858 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:56:24.858 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:56:24.858 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:56:24.859 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:56:24.859 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:56:24.859 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:56:24.859 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:56:24.859 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:24.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:24.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:56:24.860 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:56:24.860 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:56:24.860 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:56:24.860 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:56:24.861 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:56:24.861 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:56:24.861 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:56:24.861 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:56:24.861 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:56:24.861 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:24.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:24.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:56:24.862 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:56:24.862 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:56:24.862 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:56:24.862 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:56:24.862 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:56:24.862 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:56:24.863 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:56:24.863 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:56:24.863 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:56:24.863 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:24.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:24.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:56:24.863 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:56:24.864 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:56:24.864 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:56:24.864 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:56:24.864 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:56:24.864 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:56:24.864 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:56:24.864 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:56:24.865 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:56:24.865 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:24.865 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:24.865 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:56:24.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:24.865 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:56:24.865 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:56:24.866 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:56:24.866 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:24.866 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:56:24.867 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:56:29.868 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:56:29.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:56:29.870 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:56:29.910 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:56:29.911 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:56:30.282 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:56:30.282 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:56:30.282 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:56:30.282 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:56:30.283 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:56:30.283 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:56:30.283 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:56:30.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:30.283 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:56:30.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:56:30.283 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:56:30.284 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:56:30.284 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:56:30.284 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:56:30.284 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:56:30.284 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:56:30.284 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:56:30.284 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:56:30.285 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:56:30.285 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:30.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:30.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:56:30.285 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:56:30.285 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:56:30.285 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:56:30.285 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:56:30.286 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:56:30.286 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:56:30.286 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:56:30.286 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:56:30.286 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:56:30.286 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:30.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:30.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:56:30.287 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:56:30.287 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:56:30.287 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:56:30.287 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:56:30.288 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:56:30.288 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:56:30.288 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:56:30.288 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:56:30.288 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:56:30.288 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:30.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:30.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:56:30.289 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:56:30.289 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:56:30.289 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:56:30.289 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:56:30.289 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:56:30.290 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:56:30.290 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:56:30.290 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:56:30.290 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:56:30.290 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:30.290 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:30.290 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:56:30.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:30.291 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:56:30.291 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:56:30.291 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:56:30.291 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:30.292 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:56:30.293 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:56:35.294 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:56:35.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:56:35.296 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:56:35.335 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:56:35.335 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:56:35.703 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:56:35.703 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:56:35.703 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:56:35.703 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:56:35.703 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:56:35.704 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:56:35.704 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:56:35.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:35.704 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:56:35.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:56:35.704 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:56:35.704 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:56:35.705 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:56:35.705 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:56:35.705 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:56:35.705 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:56:35.705 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:56:35.705 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:56:35.705 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:56:35.705 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:35.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:35.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:56:35.706 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:56:35.707 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:56:35.707 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:56:35.707 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:56:35.707 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:56:35.707 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:56:35.707 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:56:35.708 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:56:35.708 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:56:35.708 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:35.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:35.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:56:35.708 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:56:35.708 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:56:35.708 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:56:35.709 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:56:35.709 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:56:35.709 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:56:35.709 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:56:35.709 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:56:35.709 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:56:35.709 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:35.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:35.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:56:35.710 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:56:35.710 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:56:35.710 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:56:35.710 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:56:35.710 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:56:35.711 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:56:35.711 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:56:35.711 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:56:35.711 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:56:35.711 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:35.711 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:35.711 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:56:35.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:35.712 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:56:35.712 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:56:35.712 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:56:35.712 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:35.712 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:56:35.713 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:56:40.715 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:56:40.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:56:40.716 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:56:40.757 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:56:40.758 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:56:41.117 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:56:41.118 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:56:41.118 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:56:41.118 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:56:41.118 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:56:41.118 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:56:41.118 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:56:41.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:41.119 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:56:41.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:56:41.119 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:56:41.119 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:56:41.119 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:56:41.119 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:56:41.120 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:56:41.120 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:56:41.120 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:56:41.120 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:56:41.120 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:56:41.120 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:41.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:41.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:56:41.121 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:56:41.121 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:56:41.121 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:56:41.121 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:56:41.121 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:56:41.121 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:56:41.121 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:56:41.122 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:56:41.122 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:56:41.122 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:41.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:41.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:56:41.122 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:56:41.122 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:56:41.123 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:56:41.123 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:56:41.123 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:56:41.123 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:56:41.123 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:56:41.123 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:56:41.123 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:56:41.124 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:41.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:41.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:56:41.124 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:56:41.124 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:56:41.124 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:56:41.124 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:56:41.124 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:56:41.125 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:56:41.125 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:56:41.125 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:56:41.125 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:56:41.125 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:41.125 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:41.125 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:56:41.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:41.126 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:56:41.126 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:56:41.126 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:56:41.126 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:41.126 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:56:41.127 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:56:46.129 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:56:46.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:56:46.130 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:56:46.172 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:56:46.172 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:56:46.540 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:56:46.540 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:56:46.541 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:56:46.541 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:56:46.541 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:56:46.541 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:56:46.541 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:56:46.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:46.542 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:56:46.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:56:46.542 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:56:46.542 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:56:46.542 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:56:46.542 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:56:46.542 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:56:46.543 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:56:46.543 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:56:46.543 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:56:46.543 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:56:46.543 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:46.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:46.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:56:46.544 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:56:46.544 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:56:46.544 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:56:46.544 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:56:46.544 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:56:46.544 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:56:46.544 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:56:46.545 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:56:46.545 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:56:46.545 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:46.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:46.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:56:46.545 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:56:46.545 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:56:46.545 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:56:46.546 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:56:46.546 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:56:46.546 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:56:46.546 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:56:46.546 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:56:46.546 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:56:46.546 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:46.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:46.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:56:46.547 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:56:46.547 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:56:46.547 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:56:46.547 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:56:46.547 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:56:46.548 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:56:46.548 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:56:46.548 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:56:46.548 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:56:46.548 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:46.548 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:46.548 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:56:46.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:46.549 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:56:46.549 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:56:46.549 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:56:46.549 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:46.549 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:56:46.550 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:56:51.551 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:56:51.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:56:51.553 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:56:51.593 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:56:51.594 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:56:51.950 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:56:51.951 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:56:51.951 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:56:51.951 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:56:51.951 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:56:51.951 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:56:51.951 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:56:51.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:51.952 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:56:51.952 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:56:51.952 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:56:51.952 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:56:51.952 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:56:51.952 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:56:51.953 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:56:51.953 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:56:51.953 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:56:51.953 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:56:51.953 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:56:51.953 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:51.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:51.953 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:56:51.954 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:56:51.954 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:56:51.954 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:56:51.954 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:56:51.954 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:56:51.954 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:56:51.954 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:56:51.955 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:56:51.955 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:56:51.955 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:51.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:51.955 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:56:51.955 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:56:51.955 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:56:51.956 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:56:51.956 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:56:51.956 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:56:51.956 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:56:51.956 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:56:51.956 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:56:51.956 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:56:51.957 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:51.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:51.957 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:56:51.957 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:56:51.957 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:56:51.957 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:56:51.957 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:56:51.957 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:56:51.958 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:56:51.958 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:56:51.958 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:56:51.958 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:56:51.958 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:51.958 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:51.958 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:56:51.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:51.959 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:56:51.959 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:56:51.959 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:56:51.959 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:51.959 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:56:51.960 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:56:56.961 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:56:56.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:56:56.962 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:56:57.002 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:56:57.003 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:56:57.367 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:56:57.367 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:56:57.367 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:56:57.368 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:56:57.368 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:56:57.368 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:56:57.368 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:56:57.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:57.368 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:56:57.368 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:56:57.369 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:56:57.369 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:56:57.369 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:56:57.369 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:56:57.369 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:56:57.369 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:56:57.369 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:56:57.370 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:56:57.370 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:56:57.370 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:57.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:57.370 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:56:57.370 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:56:57.370 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:56:57.371 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:56:57.371 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:56:57.371 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:56:57.371 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:56:57.371 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:56:57.371 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:56:57.371 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:56:57.371 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:57.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:57.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:56:57.372 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:56:57.372 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:56:57.372 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:56:57.372 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:56:57.372 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:56:57.373 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:56:57.373 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:56:57.373 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:56:57.373 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:56:57.373 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:57.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:57.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:56:57.373 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:56:57.374 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:56:57.374 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:56:57.374 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:56:57.374 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:56:57.374 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:56:57.374 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:56:57.374 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:56:57.375 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:56:57.375 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:57.375 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:57.375 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:56:57.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:56:57.375 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:56:57.375 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:56:57.376 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:56:57.376 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:56:57.376 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:56:57.377 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:57:02.378 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:57:02.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:57:02.379 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:57:02.430 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:57:02.430 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:57:02.697 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:57:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:57:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:57:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:57:02.699 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:57:02.699 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:57:02.699 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:57:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:57:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:57:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:57:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:57:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:57:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:57:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:57:02.702 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:57:02.702 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:57:02.702 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:57:02.702 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:57:02.703 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:02.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:02.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:57:02.703 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:57:02.704 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:57:02.704 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:57:02.704 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:57:02.704 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:57:02.705 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:57:02.705 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:57:02.705 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:57:02.705 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:57:02.705 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:02.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:02.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:57:02.706 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:57:02.707 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:57:02.707 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:57:02.707 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:57:02.707 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:57:02.708 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:57:02.708 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:57:02.708 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:57:02.708 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:57:02.709 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:02.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:02.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:57:02.709 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:57:02.709 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:57:02.710 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:57:02.710 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:57:02.710 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:57:02.710 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:57:02.711 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:57:02.711 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:57:02.711 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:57:02.711 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:02.711 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:02.712 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:57:02.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:02.712 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:57:02.713 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:57:02.713 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:57:02.713 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:02.713 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:57:02.715 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:57:07.717 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:57:07.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:57:07.718 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:57:07.759 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:57:07.759 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:57:08.121 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:57:08.122 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:57:08.122 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:57:08.122 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:57:08.122 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:57:08.122 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:57:08.122 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:57:08.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:08.123 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:57:08.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:57:08.123 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:57:08.123 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:57:08.123 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:57:08.124 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:57:08.124 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:57:08.124 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:57:08.124 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:57:08.124 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:57:08.124 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:57:08.125 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:08.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:08.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:57:08.125 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:57:08.125 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:57:08.126 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:57:08.126 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:57:08.126 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:57:08.126 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:57:08.126 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:57:08.127 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:57:08.127 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:57:08.127 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:08.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:08.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:57:08.127 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:57:08.128 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:57:08.128 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:57:08.128 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:57:08.128 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:57:08.128 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:57:08.128 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:57:08.128 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:57:08.129 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:57:08.129 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:08.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:08.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:57:08.129 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:57:08.129 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:57:08.130 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:57:08.130 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:57:08.130 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:57:08.130 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:57:08.130 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:57:08.130 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:57:08.131 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:57:08.131 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:08.131 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:08.131 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:57:08.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:08.131 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:57:08.132 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:57:08.132 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:57:08.132 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:08.132 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:57:08.133 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:57:13.134 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:57:13.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:57:13.136 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:57:13.177 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:57:13.177 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:57:13.534 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:57:13.535 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:57:13.535 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:57:13.535 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:57:13.535 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:57:13.536 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:57:13.536 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:57:13.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:13.536 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:57:13.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:57:13.536 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:57:13.537 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:57:13.537 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:57:13.537 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:57:13.537 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:57:13.537 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:57:13.538 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:57:13.538 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:57:13.538 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:57:13.538 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:13.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:13.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:57:13.539 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:57:13.539 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:57:13.540 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:57:13.540 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:57:13.540 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:57:13.540 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:57:13.541 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:57:13.541 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:57:13.541 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:57:13.541 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:13.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:13.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:57:13.542 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:57:13.542 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:57:13.543 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:57:13.543 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:57:13.543 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:57:13.543 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:57:13.544 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:57:13.544 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:57:13.544 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:57:13.544 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:13.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:13.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:57:13.545 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:57:13.545 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:57:13.546 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:57:13.546 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:57:13.546 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:57:13.546 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:57:13.547 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:57:13.547 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:57:13.547 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:57:13.547 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:13.547 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:13.547 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:57:13.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:13.548 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:57:13.548 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:57:13.548 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:57:13.548 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:13.548 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:57:13.550 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:57:18.551 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:57:18.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:57:18.584 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:57:18.626 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:57:18.626 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:57:18.991 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:57:18.992 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:57:18.992 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:57:18.992 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:57:18.992 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:57:18.992 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:57:18.993 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:57:18.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:18.993 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:57:18.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:57:18.993 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:57:18.993 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:57:18.993 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:57:18.994 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:57:18.994 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:57:18.994 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:57:18.994 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:57:18.994 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:57:18.994 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:57:18.994 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:18.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:18.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:57:18.995 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:57:18.995 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:57:18.995 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:57:18.995 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:57:18.996 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:57:18.996 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:57:18.996 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:57:18.996 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:57:18.996 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:57:18.996 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:18.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:18.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:57:18.997 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:57:18.997 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:57:18.997 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:57:18.997 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:57:18.997 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:57:18.997 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:57:18.997 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:57:18.998 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:57:18.998 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:57:18.998 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:18.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:18.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:57:18.998 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:57:18.998 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:57:18.999 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:57:18.999 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:57:18.999 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:57:18.999 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:57:18.999 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:57:18.999 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:57:18.999 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:57:19.000 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:19.000 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:19.000 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:57:19.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:19.000 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:57:19.000 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:57:19.000 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:57:19.001 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:19.001 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:57:19.002 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:57:24.003 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:57:24.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:57:24.005 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:57:24.047 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:57:24.048 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:57:24.414 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:57:24.415 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:57:24.415 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:57:24.415 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:57:24.415 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:57:24.415 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:57:24.416 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:57:24.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:24.416 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:57:24.416 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:57:24.416 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:57:24.416 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:57:24.416 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:57:24.416 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:57:24.417 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:57:24.417 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:57:24.417 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:57:24.417 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:57:24.417 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:57:24.417 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:24.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:24.418 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:57:24.418 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:57:24.418 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:57:24.418 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:57:24.418 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:57:24.418 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:57:24.418 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:57:24.419 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:57:24.419 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:57:24.419 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:57:24.419 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:24.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:24.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:57:24.419 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:57:24.420 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:57:24.420 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:57:24.420 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:57:24.421 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:57:24.421 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:57:24.421 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:57:24.421 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:57:24.421 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:57:24.421 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:24.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:24.422 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:57:24.422 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:57:24.422 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:57:24.422 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:57:24.422 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:57:24.422 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:57:24.423 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:57:24.423 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:57:24.423 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:57:24.423 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:57:24.423 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:24.423 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:24.423 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:57:24.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:24.424 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:57:24.424 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:57:24.424 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:57:24.424 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:24.424 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:57:24.425 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:57:29.426 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:57:29.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:57:29.428 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:57:29.468 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:57:29.469 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:57:29.834 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:57:29.835 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:57:29.835 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:57:29.836 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:57:29.836 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:57:29.836 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:57:29.836 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:57:29.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:29.837 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:57:29.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:57:29.837 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:57:29.837 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:57:29.838 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:57:29.838 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:57:29.838 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:57:29.839 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:57:29.839 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:57:29.839 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:57:29.839 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:57:29.840 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:29.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:29.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:57:29.841 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:57:29.841 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:57:29.841 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:57:29.841 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:57:29.841 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:57:29.841 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:57:29.841 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:57:29.842 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:57:29.842 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:57:29.842 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:29.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:29.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:57:29.842 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:57:29.842 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:57:29.843 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:57:29.843 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:57:29.843 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:57:29.843 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:57:29.843 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:57:29.843 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:57:29.843 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:57:29.844 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:29.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:29.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:57:29.844 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:57:29.844 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:57:29.845 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:57:29.845 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:57:29.845 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:57:29.845 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:57:29.846 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:57:29.846 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:57:29.846 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:57:29.846 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:29.846 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:29.846 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:57:29.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:29.847 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:57:29.847 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:57:29.847 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:57:29.847 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:29.847 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:57:29.848 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:57:34.849 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:57:34.850 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:57:34.850 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:57:34.890 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:57:34.891 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:57:35.259 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:57:35.259 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:57:35.259 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:57:35.259 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:57:35.259 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:57:35.260 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:57:35.260 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:57:35.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:35.260 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:57:35.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:57:35.260 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:57:35.260 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:57:35.261 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:57:35.261 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:57:35.261 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:57:35.261 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:57:35.261 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:57:35.261 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:57:35.261 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:57:35.262 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:35.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:35.262 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:57:35.262 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:57:35.262 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:57:35.262 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:57:35.262 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:57:35.263 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:57:35.263 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:57:35.263 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:57:35.263 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:57:35.263 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:57:35.263 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:35.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:35.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:57:35.264 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:57:35.264 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:57:35.264 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:57:35.264 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:57:35.264 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:57:35.264 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:57:35.264 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:57:35.265 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:57:35.265 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:57:35.265 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:35.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:35.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:57:35.265 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:57:35.265 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:57:35.266 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:57:35.266 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:57:35.266 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:57:35.266 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:57:35.266 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:57:35.266 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:57:35.266 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:57:35.266 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:35.267 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:35.267 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:57:35.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:35.267 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:57:35.267 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:57:35.267 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:57:35.267 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:35.268 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:57:35.269 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:57:40.270 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:57:40.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:57:40.272 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:57:40.315 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:57:40.316 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:57:40.682 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:57:40.683 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:57:40.683 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:57:40.683 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:57:40.683 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:57:40.683 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:57:40.683 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:57:40.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:40.684 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:57:40.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:57:40.684 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:57:40.684 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:57:40.684 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:57:40.684 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:57:40.685 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:57:40.685 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:57:40.685 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:57:40.685 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:57:40.685 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:57:40.685 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:40.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:40.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:57:40.686 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:57:40.688 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:57:40.688 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:57:40.688 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:57:40.689 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:57:40.689 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:57:40.689 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:57:40.690 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:57:40.690 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:57:40.690 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:40.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:40.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:57:40.691 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:57:40.691 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:57:40.692 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:57:40.692 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:57:40.692 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:57:40.692 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:57:40.693 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:57:40.694 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:57:40.694 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:57:40.695 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:40.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:40.695 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:57:40.695 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:57:40.696 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:57:40.696 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:57:40.696 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:57:40.696 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:57:40.697 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:57:40.697 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:57:40.697 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:57:40.698 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:57:40.698 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:40.698 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:40.698 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:57:40.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:40.699 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:57:40.699 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:57:40.700 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:57:40.700 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:40.700 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:57:40.702 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:57:45.703 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:57:45.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:57:45.704 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:57:45.745 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:57:45.746 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:57:46.107 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:57:46.107 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:57:46.107 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:57:46.107 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:57:46.108 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:57:46.108 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:57:46.108 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:57:46.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:46.108 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:57:46.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:57:46.108 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:57:46.109 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:57:46.109 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:57:46.109 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:57:46.109 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:57:46.109 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:57:46.109 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:57:46.109 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:57:46.110 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:57:46.110 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:46.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:46.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:57:46.110 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:57:46.110 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:57:46.110 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:57:46.111 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:57:46.111 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:57:46.111 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:57:46.112 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:57:46.112 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:57:46.112 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:57:46.112 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:46.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:46.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:57:46.113 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:57:46.114 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:57:46.114 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:57:46.114 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:57:46.114 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:57:46.115 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:57:46.115 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:57:46.115 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:57:46.116 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:57:46.116 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:46.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:46.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:57:46.117 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:57:46.117 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:57:46.117 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:57:46.117 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:57:46.118 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:57:46.118 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:57:46.118 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:57:46.119 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:57:46.119 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:57:46.119 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:46.119 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:46.119 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:57:46.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:46.120 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:57:46.120 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:57:46.121 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:57:46.121 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:46.121 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:57:46.123 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:57:51.124 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:57:51.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:57:51.126 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:57:51.169 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:57:51.169 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:57:51.533 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:57:51.534 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:57:51.534 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:57:51.534 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:57:51.534 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:57:51.534 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:57:51.535 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:57:51.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:51.535 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:57:51.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:57:51.535 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:57:51.535 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:57:51.535 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:57:51.536 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:57:51.536 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:57:51.536 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:57:51.536 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:57:51.536 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:57:51.536 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:57:51.536 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:51.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:51.537 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:57:51.537 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:57:51.537 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:57:51.537 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:57:51.537 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:57:51.537 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:57:51.538 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:57:51.538 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:57:51.538 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:57:51.538 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:57:51.538 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:51.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:51.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:57:51.539 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:57:51.539 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:57:51.539 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:57:51.539 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:57:51.539 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:57:51.539 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:57:51.539 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:57:51.539 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:57:51.540 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:57:51.540 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:51.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:51.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:57:51.540 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:57:51.540 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:57:51.540 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:57:51.541 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:57:51.542 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:57:51.542 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:57:51.542 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:57:51.542 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:57:51.543 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:57:51.543 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:51.543 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:51.543 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:57:51.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:51.543 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:57:51.543 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:57:51.544 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:57:51.544 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:51.544 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:57:51.545 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:57:56.546 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:57:56.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:57:56.547 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:57:56.589 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:57:56.589 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:57:56.957 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:57:56.957 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:57:56.957 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:57:56.958 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:57:56.958 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:57:56.958 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:57:56.958 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:57:56.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:56.958 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:57:56.958 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:57:56.959 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:57:56.959 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:57:56.959 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:57:56.959 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:57:56.959 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:57:56.959 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:57:56.960 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:57:56.960 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:57:56.960 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:57:56.960 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:56.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:56.960 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:57:56.961 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:57:56.961 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:57:56.961 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:57:56.961 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:57:56.961 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:57:56.961 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:57:56.961 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:57:56.962 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:57:56.962 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:57:56.962 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:56.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:56.962 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:57:56.962 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:57:56.962 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:57:56.963 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:57:56.963 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:57:56.963 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:57:56.963 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:57:56.963 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:57:56.963 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:57:56.964 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:57:56.964 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:56.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:56.964 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:57:56.964 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:57:56.964 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:57:56.964 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:57:56.965 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:57:56.965 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:57:56.965 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:57:56.965 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:57:56.965 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:57:56.965 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:57:56.966 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:56.966 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:56.966 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:57:56.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:57:56.967 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:57:56.967 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:57:56.967 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:57:56.967 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:57:56.967 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:57:56.968 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:58:01.969 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:58:01.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:58:01.971 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:58:02.012 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:58:02.013 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:58:02.378 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:58:02.378 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:58:02.379 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:58:02.379 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:58:02.379 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:58:02.379 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:58:02.379 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:58:02.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:02.379 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:58:02.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:02.380 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:58:02.380 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:02.380 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:02.380 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:58:02.380 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:02.380 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:58:02.381 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:02.381 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:58:02.381 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:58:02.381 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:02.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:02.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:02.382 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:58:02.382 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:02.382 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:02.382 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:58:02.382 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:02.382 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:58:02.383 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:02.383 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:58:02.383 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:58:02.383 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:02.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:02.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:02.384 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:58:02.384 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:02.384 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:02.384 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:58:02.384 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:02.384 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:58:02.384 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:02.385 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:58:02.385 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:58:02.385 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:02.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:02.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:02.385 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:58:02.385 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:02.385 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:02.386 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:58:02.386 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:02.386 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:58:02.386 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:02.386 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:58:02.386 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:58:02.386 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:02.387 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:02.387 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:58:02.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:02.387 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:58:02.387 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:58:02.387 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:58:02.387 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:02.387 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:58:02.388 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:58:07.389 INFO:tasks.rgw_multisite_tests:test s3 notification set/get/delete on master ... FAIL 2020-11-09T18:58:07.390 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_objcet_timing ... ok 2020-11-09T18:58:07.391 INFO:tasks.rgw_multisite_tests:test pushing amqp s3 notification on master ... ok 2020-11-09T18:58:07.391 INFO:tasks.rgw_multisite_tests:test reservation cleanup after gateway crash ... ok 2020-11-09T18:58:07.392 INFO:tasks.rgw_multisite_tests:test gateway recovery of persistent notifications ... ok 2020-11-09T18:58:07.392 INFO:tasks.rgw_multisite_tests:test pushing persistent notification via two gateways ... ok 2020-11-09T18:58:07.393 INFO:tasks.rgw_multisite_tests:test pushing persistent notification when one of the endpoints has error ... ok 2020-11-09T18:58:07.393 INFO:tasks.rgw_multisite_tests:test pushing persistent notification http ... ok 2020-11-09T18:58:07.393 INFO:tasks.rgw_multisite_tests:test pushing persistent notification amqp ... ok 2020-11-09T18:58:07.394 INFO:tasks.rgw_multisite_tests:test pushing persistent notification of large notifications ... ok 2020-11-09T18:58:07.394 INFO:tasks.rgw_multisite_tests:test pushing persistent notification pushback ... ok 2020-11-09T18:58:07.395 INFO:tasks.rgw_multisite_tests:test pushing kafka s3 notification on master ... ok 2020-11-09T18:58:07.395 INFO:tasks.rgw_multisite_tests:test pushing kafka s3 notification on master ... ok 2020-11-09T18:58:07.396 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_ps_s3_notification_push_kafka_security_ssl ... ok 2020-11-09T18:58:07.396 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_ps_s3_notification_push_kafka_security_ssl_sasl ... ok 2020-11-09T18:58:07.396 INFO:tasks.rgw_multisite_tests:test deletion of multiple keys on master ... ok 2020-11-09T18:58:07.397 INFO:tasks.rgw_multisite_tests:test pushing http s3 notification on master ... ok 2020-11-09T18:58:07.397 INFO:tasks.rgw_multisite_tests:test that opaque id set in topic, is sent in notification ... ok 2020-11-09T18:58:07.398 INFO:tasks.rgw_multisite_tests:test that opaque id set in topic, is sent in notification on master ... ok 2020-11-09T18:58:07.398 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:58:07.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:58:07.399 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:58:07.443 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:58:07.444 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:58:07.656 INFO:teuthology.orchestra.run.smithi051.stderr:No --period given, using current period=f6258aba-300a-4d5d-ad93-a7c198a0e0a1 2020-11-09T18:58:07.690 INFO:teuthology.orchestra.run.smithi051.stdout:[ 2020-11-09T18:58:07.691 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:58:07.691 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604946340.774139_451.1", 2020-11-09T18:58:07.691 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "2020-11-09T18:25:40.774139Z" 2020-11-09T18:58:07.691 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:58:07.691 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:58:07.691 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:58:07.692 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:58:07.692 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:58:07.692 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:58:07.692 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:58:07.692 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:58:07.692 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:58:07.692 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:58:07.693 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:58:07.693 INFO:teuthology.orchestra.run.smithi051.stdout: "last_update": "0.000000" 2020-11-09T18:58:07.693 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:58:07.693 INFO:teuthology.orchestra.run.smithi051.stdout:] 2020-11-09T18:58:07.694 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-09T18:58:07.694 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:58:07.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:58:07.695 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:58:07.737 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:58:07.737 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:58:08.019 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:58:08.020 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:58:08.020 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:58:08.020 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:58:08.020 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:58:08.020 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:58:08.021 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:58:08.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:08.021 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:58:08.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:08.021 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:58:08.021 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:08.021 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:08.022 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946340.774139_451.1", 2020-11-09T18:58:08.022 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:08.022 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:58:08.022 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:08.022 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:40.774139Z", 2020-11-09T18:58:08.022 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:58:08.022 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:08.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:08.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:08.023 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:58:08.023 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:08.023 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:08.023 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:58:08.023 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:08.024 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:58:08.024 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:08.024 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:58:08.024 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:58:08.024 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:08.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:08.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:08.025 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:58:08.025 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:08.025 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:08.025 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:58:08.025 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:08.026 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:58:08.026 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:08.026 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:58:08.026 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:58:08.026 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:08.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:08.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:08.027 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:58:08.027 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:08.027 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:08.027 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:58:08.027 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:08.027 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:58:08.027 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:08.028 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:58:08.028 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:58:08.029 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:08.029 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:08.029 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:58:08.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:08.029 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:58:08.029 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:58:08.029 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:58:08.030 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:08.030 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:58:08.031 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-09T18:58:08.031 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-09T18:58:08.032 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:58:08.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:58:08.033 INFO:teuthology.orchestra.run.smithi051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:58:08.075 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:58:08.075 INFO:teuthology.orchestra.run.smithi051.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:58:08.334 INFO:teuthology.orchestra.run.smithi051.stdout:{ 2020-11-09T18:58:08.335 INFO:teuthology.orchestra.run.smithi051.stdout: "sync_status": { 2020-11-09T18:58:08.335 INFO:teuthology.orchestra.run.smithi051.stdout: "info": { 2020-11-09T18:58:08.335 INFO:teuthology.orchestra.run.smithi051.stdout: "status": "sync", 2020-11-09T18:58:08.336 INFO:teuthology.orchestra.run.smithi051.stdout: "num_shards": 4, 2020-11-09T18:58:08.336 INFO:teuthology.orchestra.run.smithi051.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:58:08.336 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 4 2020-11-09T18:58:08.337 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:58:08.337 INFO:teuthology.orchestra.run.smithi051.stdout: "markers": [ 2020-11-09T18:58:08.337 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:58:08.337 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 0, 2020-11-09T18:58:08.338 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:58:08.338 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:58:08.338 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604946340.774139_451.1", 2020-11-09T18:58:08.338 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:58:08.339 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 1, 2020-11-09T18:58:08.339 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:58:08.339 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:25:40.774139Z", 2020-11-09T18:58:08.341 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 4 2020-11-09T18:58:08.341 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:58:08.341 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:58:08.341 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:58:08.342 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 1, 2020-11-09T18:58:08.342 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:58:08.344 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:58:08.345 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:58:08.347 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:58:08.348 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:58:08.349 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:58:08.350 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:58:08.351 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:58:08.351 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:58:08.351 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:58:08.351 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:58:08.351 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 2, 2020-11-09T18:58:08.351 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:58:08.352 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:58:08.352 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:58:08.352 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:58:08.352 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:58:08.352 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:58:08.352 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:58:08.353 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:58:08.353 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:58:08.353 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:58:08.353 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-11-09T18:58:08.353 INFO:teuthology.orchestra.run.smithi051.stdout: "key": 3, 2020-11-09T18:58:08.353 INFO:teuthology.orchestra.run.smithi051.stdout: "val": { 2020-11-09T18:58:08.353 INFO:teuthology.orchestra.run.smithi051.stdout: "state": 1, 2020-11-09T18:58:08.354 INFO:teuthology.orchestra.run.smithi051.stdout: "marker": "", 2020-11-09T18:58:08.354 INFO:teuthology.orchestra.run.smithi051.stdout: "next_step_marker": "", 2020-11-09T18:58:08.354 INFO:teuthology.orchestra.run.smithi051.stdout: "total_entries": 0, 2020-11-09T18:58:08.354 INFO:teuthology.orchestra.run.smithi051.stdout: "pos": 0, 2020-11-09T18:58:08.354 INFO:teuthology.orchestra.run.smithi051.stdout: "timestamp": "0.000000", 2020-11-09T18:58:08.354 INFO:teuthology.orchestra.run.smithi051.stdout: "realm_epoch": 2 2020-11-09T18:58:08.355 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:58:08.355 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:58:08.355 INFO:teuthology.orchestra.run.smithi051.stdout: ] 2020-11-09T18:58:08.355 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-11-09T18:58:08.355 INFO:teuthology.orchestra.run.smithi051.stdout: "full_sync": { 2020-11-09T18:58:08.355 INFO:teuthology.orchestra.run.smithi051.stdout: "total": 1, 2020-11-09T18:58:08.355 INFO:teuthology.orchestra.run.smithi051.stdout: "complete": 1 2020-11-09T18:58:08.356 INFO:teuthology.orchestra.run.smithi051.stdout: } 2020-11-09T18:58:08.356 INFO:teuthology.orchestra.run.smithi051.stdout:} 2020-11-09T18:58:08.357 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-09T18:58:08.357 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-09T18:58:08.358 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:58:08.358 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:58:08.359 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:58:08.406 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:58:08.406 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:58:08.669 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:58:08.669 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:58:08.669 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:58:08.669 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:58:08.670 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:58:08.670 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:58:08.670 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:58:08.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:08.670 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:58:08.670 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:08.670 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:58:08.671 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:08.671 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:08.671 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:58:08.671 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:08.671 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:58:08.671 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:08.671 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:58:08.672 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:58:08.672 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:08.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:08.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:08.672 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:58:08.672 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:08.672 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:08.673 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:58:08.673 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:08.673 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:58:08.673 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:08.673 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:58:08.673 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:58:08.673 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:08.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:08.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:08.674 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:58:08.674 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:08.674 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:08.674 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:58:08.674 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:08.674 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:58:08.675 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:08.675 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:58:08.675 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:58:08.675 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:08.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:08.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:08.675 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:58:08.676 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:08.676 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:08.676 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:58:08.676 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:08.676 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:58:08.676 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:08.677 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:58:08.677 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:58:08.677 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:08.678 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:08.678 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:58:08.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:08.678 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:58:08.678 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:58:08.679 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:58:08.679 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:08.679 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:58:08.680 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:58:13.681 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:58:13.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:58:13.682 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:58:13.721 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:58:13.722 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:58:14.188 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:58:14.188 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:58:14.188 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:58:14.189 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:58:14.189 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:58:14.189 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:58:14.189 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:58:14.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:14.189 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:58:14.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:14.190 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:58:14.190 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:14.190 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:14.190 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:58:14.190 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:14.190 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:58:14.191 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:14.191 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:58:14.191 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:58:14.191 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:14.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:14.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:14.191 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:58:14.192 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:14.192 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:14.192 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:58:14.192 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:14.192 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:58:14.192 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:14.192 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:58:14.192 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:58:14.193 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:14.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:14.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:14.193 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:58:14.193 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:14.193 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:14.193 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:58:14.194 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:14.194 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:58:14.194 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:14.194 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:58:14.194 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:58:14.194 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:14.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:14.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:14.195 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:58:14.195 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:14.195 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:14.195 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:58:14.195 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:14.195 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:58:14.196 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:14.196 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:58:14.196 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:58:14.196 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:14.196 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:14.196 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:58:14.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:14.197 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:58:14.197 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:58:14.197 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:58:14.197 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:14.197 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:58:14.199 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:58:19.200 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:58:19.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:58:19.202 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:58:19.242 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:58:19.243 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:58:19.611 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:58:19.612 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:58:19.612 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:58:19.612 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:58:19.612 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:58:19.612 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:58:19.613 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:58:19.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:19.613 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:58:19.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:19.613 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:58:19.613 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:19.614 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:19.614 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:58:19.614 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:19.614 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:58:19.614 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:19.614 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:58:19.615 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:58:19.615 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:19.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:19.615 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:19.615 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:58:19.615 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:19.615 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:19.616 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:58:19.616 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:19.616 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:58:19.616 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:19.616 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:58:19.616 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:58:19.617 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:19.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:19.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:19.617 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:58:19.617 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:19.617 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:19.617 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:58:19.618 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:19.618 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:58:19.618 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:19.618 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:58:19.618 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:58:19.618 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:19.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:19.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:19.619 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:58:19.619 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:19.619 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:19.619 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:58:19.619 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:19.619 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:58:19.620 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:19.620 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:58:19.620 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:58:19.621 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:19.622 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:19.622 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:58:19.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:19.622 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:58:19.622 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:58:19.622 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:58:19.623 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:19.623 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:58:19.624 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:58:24.625 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:58:24.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:58:24.626 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:58:24.668 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:58:24.668 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:58:25.131 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:58:25.132 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:58:25.132 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:58:25.132 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:58:25.132 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:58:25.132 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:58:25.133 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:58:25.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:25.133 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:58:25.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:25.133 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:58:25.133 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:25.133 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:25.134 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:58:25.134 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:25.134 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:58:25.134 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:25.134 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:58:25.134 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:58:25.134 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:25.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:25.135 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:25.135 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:58:25.135 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:25.135 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:25.135 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:58:25.135 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:25.135 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:58:25.136 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:25.136 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:58:25.136 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:58:25.136 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:25.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:25.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:25.136 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:58:25.137 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:25.137 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:25.137 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:58:25.137 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:25.137 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:58:25.137 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:25.137 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:58:25.138 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:58:25.138 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:25.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:25.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:25.138 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:58:25.138 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:25.138 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:25.139 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:58:25.139 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:25.139 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:58:25.139 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:25.139 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:58:25.139 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:58:25.140 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:25.140 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:25.140 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:58:25.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:25.140 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:58:25.140 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:58:25.141 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:58:25.141 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:25.141 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:58:25.142 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:58:30.144 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:58:30.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:58:30.145 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:58:30.188 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:58:30.188 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:58:30.558 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:58:30.558 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:58:30.558 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:58:30.559 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:58:30.559 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:58:30.559 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:58:30.559 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:58:30.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:30.559 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:58:30.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:30.560 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:58:30.560 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:30.560 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:30.560 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:58:30.560 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:30.560 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:58:30.560 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:30.561 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:58:30.561 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:58:30.561 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:30.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:30.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:30.561 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:58:30.561 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:30.561 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:30.562 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:58:30.562 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:30.562 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:58:30.562 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:30.562 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:58:30.562 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:58:30.563 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:30.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:30.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:30.563 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:58:30.563 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:30.563 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:30.563 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:58:30.564 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:30.564 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:58:30.564 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:30.564 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:58:30.564 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:58:30.564 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:30.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:30.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:30.565 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:58:30.565 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:30.565 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:30.565 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:58:30.565 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:30.565 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:58:30.565 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:30.566 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:58:30.567 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:58:30.567 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:30.567 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:30.567 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:58:30.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:30.567 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:58:30.568 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:58:30.568 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:58:30.568 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:30.568 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:58:30.569 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:58:35.572 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:58:35.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:58:35.574 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:58:35.619 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:58:35.619 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:58:35.965 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:58:35.966 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:58:35.966 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:58:35.966 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:58:35.966 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:58:35.966 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:58:35.967 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:58:35.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:35.967 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:58:35.967 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:35.967 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:58:35.967 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:35.967 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:35.968 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:58:35.968 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:35.968 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:58:35.975 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:35.975 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:58:35.976 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:58:35.976 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:35.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:35.976 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:35.976 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:58:35.976 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:35.977 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:35.977 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:58:35.977 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:35.977 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:58:35.977 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:35.977 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:58:35.977 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:58:35.977 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:35.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:35.978 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:35.978 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:58:35.978 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:35.978 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:35.978 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:58:35.978 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:35.979 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:58:35.979 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:35.979 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:58:35.979 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:58:35.979 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:35.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:35.979 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:35.980 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:58:35.981 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:35.981 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:35.981 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:58:35.981 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:35.981 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:58:35.981 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:35.981 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:58:35.982 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:58:35.982 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:35.983 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:35.985 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:58:35.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:35.985 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:58:35.985 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:58:35.985 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:58:35.985 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:35.985 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:58:35.987 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:58:40.988 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:58:40.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:58:40.991 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:58:41.038 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:58:41.039 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:58:41.408 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:58:41.408 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:58:41.409 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:58:41.409 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:58:41.409 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:58:41.410 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:58:41.411 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:58:41.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:41.412 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:58:41.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:41.412 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:58:41.413 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:41.413 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:41.413 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:58:41.414 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:41.415 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:58:41.415 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:41.415 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:58:41.416 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:58:41.416 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:41.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:41.418 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:41.419 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:58:41.419 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:41.419 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:41.419 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:58:41.420 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:41.420 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:58:41.420 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:41.420 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:58:41.420 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:58:41.420 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:41.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:41.421 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:41.421 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:58:41.421 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:41.421 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:41.421 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:58:41.421 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:41.421 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:58:41.422 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:41.422 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:58:41.422 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:58:41.422 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:41.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:41.422 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:41.422 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:58:41.423 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:41.423 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:41.423 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:58:41.423 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:41.423 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:58:41.423 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:41.423 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:58:41.424 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:58:41.424 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:41.424 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:41.424 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:58:41.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:41.424 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:58:41.424 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:58:41.425 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:58:41.425 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:41.425 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:58:41.426 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:58:46.427 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:58:46.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:58:46.428 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:58:46.469 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:58:46.470 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:58:46.835 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:58:46.836 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:58:46.836 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:58:46.836 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:58:46.836 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:58:46.837 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:58:46.837 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:58:46.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:46.839 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:58:46.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:46.839 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:58:46.840 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:46.840 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:46.840 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:58:46.840 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:46.841 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:58:46.841 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:46.841 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:58:46.842 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:58:46.842 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:46.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:46.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:46.843 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:58:46.843 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:46.844 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:46.845 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:58:46.845 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:46.846 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:58:46.846 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:46.846 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:58:46.847 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:58:46.847 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:46.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:46.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:46.847 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:58:46.847 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:46.847 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:46.848 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:58:46.848 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:46.848 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:58:46.848 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:46.849 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:58:46.849 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:58:46.849 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:46.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:46.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:46.849 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:58:46.849 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:46.850 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:46.850 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:58:46.850 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:46.850 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:58:46.850 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:46.850 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:58:46.850 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:58:46.851 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:46.851 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:46.851 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:58:46.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:46.851 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:58:46.851 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:58:46.851 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:58:46.852 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:46.852 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:58:46.853 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:58:51.854 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:58:51.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:58:51.855 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:58:51.896 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:58:51.896 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:58:52.264 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:58:52.264 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:58:52.264 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:58:52.265 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:58:52.265 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:58:52.265 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:58:52.265 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:58:52.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:52.265 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:58:52.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:52.266 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:58:52.266 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:52.266 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:52.266 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:58:52.266 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:52.266 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:58:52.267 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:52.267 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:58:52.267 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:58:52.267 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:52.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:52.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:52.267 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:58:52.267 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:52.268 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:52.268 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:58:52.268 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:52.268 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:58:52.268 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:52.268 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:58:52.268 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:58:52.269 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:52.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:52.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:52.269 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:58:52.269 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:52.269 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:52.269 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:58:52.270 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:52.270 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:58:52.270 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:52.270 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:58:52.270 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:58:52.270 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:52.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:52.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:52.271 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:58:52.271 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:52.271 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:52.271 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:58:52.271 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:52.271 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:58:52.272 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:52.272 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:58:52.272 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:58:52.272 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:52.273 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:52.273 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:58:52.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:52.273 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:58:52.274 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:58:52.274 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:58:52.274 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:52.274 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:58:52.275 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:58:57.277 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:58:57.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:58:57.280 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:58:57.321 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:58:57.322 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:58:57.691 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:58:57.691 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:58:57.691 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:58:57.691 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:58:57.692 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:58:57.692 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:58:57.692 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:58:57.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:57.692 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:58:57.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:57.692 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:58:57.693 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:57.693 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:57.693 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:58:57.694 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:57.694 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:58:57.694 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:57.694 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:58:57.694 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:58:57.694 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:57.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:57.695 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:57.695 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:58:57.695 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:57.695 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:57.695 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:58:57.695 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:57.695 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:58:57.696 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:57.696 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:58:57.696 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:58:57.696 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:57.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:57.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:57.696 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:58:57.697 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:57.697 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:57.697 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:58:57.697 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:57.697 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:58:57.697 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:57.697 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:58:57.698 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:58:57.698 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:57.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:57.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:58:57.698 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:58:57.699 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:58:57.699 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:58:57.699 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:58:57.699 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:58:57.699 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:58:57.699 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:58:57.701 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:58:57.702 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:58:57.702 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:57.702 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:57.702 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:58:57.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:58:57.703 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:58:57.703 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:58:57.703 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:58:57.703 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:58:57.703 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:58:57.704 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:59:02.706 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:59:02.706 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:59:02.707 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:59:02.748 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:59:02.748 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:59:03.114 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:59:03.115 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:59:03.115 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:59:03.115 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:59:03.115 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:59:03.115 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:59:03.115 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:59:03.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:03.116 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:59:03.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:59:03.116 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:59:03.116 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:59:03.116 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:59:03.116 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:59:03.116 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:59:03.117 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:59:03.117 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:59:03.117 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:59:03.117 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:59:03.117 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:03.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:03.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:59:03.118 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:59:03.118 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:59:03.118 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:59:03.118 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:59:03.118 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:59:03.118 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:59:03.119 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:59:03.119 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:59:03.119 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:59:03.119 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:03.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:03.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:59:03.119 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:59:03.120 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:59:03.120 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:59:03.120 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:59:03.120 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:59:03.120 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:59:03.120 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:59:03.120 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:59:03.120 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:59:03.121 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:03.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:03.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:59:03.121 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:59:03.121 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:59:03.121 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:59:03.121 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:59:03.122 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:59:03.122 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:59:03.122 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:59:03.122 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:59:03.122 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:59:03.122 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:03.122 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:03.123 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:59:03.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:03.123 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:59:03.123 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:59:03.123 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:59:03.123 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:03.123 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:59:03.124 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:59:08.126 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:59:08.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:59:08.128 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:59:08.171 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:59:08.171 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:59:08.531 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:59:08.531 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:59:08.531 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:59:08.531 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:59:08.532 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:59:08.532 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:59:08.532 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:59:08.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:08.532 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:59:08.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:59:08.533 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:59:08.533 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:59:08.533 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:59:08.533 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:59:08.533 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:59:08.533 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:59:08.533 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:59:08.534 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:59:08.534 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:59:08.534 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:08.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:08.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:59:08.534 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:59:08.535 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:59:08.535 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:59:08.535 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:59:08.535 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:59:08.535 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:59:08.535 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:59:08.535 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:59:08.535 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:59:08.536 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:08.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:08.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:59:08.536 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:59:08.536 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:59:08.536 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:59:08.537 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:59:08.537 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:59:08.537 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:59:08.537 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:59:08.537 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:59:08.537 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:59:08.537 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:08.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:08.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:59:08.538 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:59:08.538 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:59:08.538 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:59:08.538 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:59:08.538 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:59:08.538 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:59:08.539 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:59:08.539 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:59:08.539 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:59:08.539 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:08.539 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:08.539 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:59:08.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:08.540 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:59:08.540 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:59:08.540 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:59:08.540 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:08.540 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:59:08.541 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:59:13.543 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:59:13.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:59:13.544 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:59:13.585 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:59:13.586 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:59:13.948 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:59:13.948 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:59:13.948 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:59:13.948 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:59:13.949 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:59:13.949 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:59:13.949 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:59:13.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:13.949 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:59:13.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:59:13.950 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:59:13.950 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:59:13.950 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:59:13.950 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:59:13.950 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:59:13.950 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:59:13.951 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:59:13.951 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:59:13.951 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:59:13.951 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:13.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:13.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:59:13.951 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:59:13.952 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:59:13.952 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:59:13.952 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:59:13.952 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:59:13.952 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:59:13.952 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:59:13.952 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:59:13.953 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:59:13.953 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:13.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:13.953 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:59:13.953 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:59:13.953 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:59:13.954 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:59:13.954 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:59:13.954 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:59:13.954 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:59:13.954 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:59:13.954 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:59:13.955 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:59:13.955 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:13.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:13.955 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:59:13.955 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:59:13.955 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:59:13.956 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:59:13.956 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:59:13.956 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:59:13.956 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:59:13.956 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:59:13.957 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:59:13.957 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:59:13.957 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:13.957 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:13.957 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:59:13.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:13.958 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:59:13.958 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:59:13.958 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:59:13.958 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:13.958 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:59:13.959 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:59:18.961 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:59:18.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:59:18.962 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:59:19.002 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:59:19.003 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:59:19.375 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:59:19.376 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:59:19.376 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:59:19.376 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:59:19.376 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:59:19.376 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:59:19.377 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:59:19.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:19.377 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:59:19.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:59:19.377 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:59:19.377 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:59:19.377 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:59:19.378 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:59:19.378 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:59:19.378 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:59:19.378 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:59:19.378 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:59:19.378 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:59:19.378 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:19.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:19.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:59:19.379 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:59:19.379 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:59:19.379 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:59:19.380 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:59:19.380 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:59:19.380 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:59:19.380 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:59:19.380 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:59:19.381 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:59:19.381 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:19.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:19.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:59:19.381 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:59:19.381 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:59:19.381 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:59:19.382 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:59:19.382 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:59:19.382 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:59:19.382 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:59:19.382 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:59:19.382 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:59:19.382 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:19.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:19.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:59:19.383 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:59:19.383 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:59:19.383 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:59:19.383 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:59:19.383 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:59:19.384 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:59:19.384 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:59:19.384 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:59:19.384 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:59:19.384 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:19.384 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:19.384 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:59:19.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:19.385 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:59:19.385 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:59:19.385 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:59:19.385 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:19.385 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:59:19.386 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:59:24.387 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:59:24.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:59:24.388 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:59:24.429 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:59:24.430 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:59:24.691 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:59:24.692 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:59:24.692 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:59:24.692 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:59:24.692 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:59:24.692 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:59:24.692 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:59:24.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:24.693 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:59:24.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:59:24.693 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:59:24.693 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:59:24.693 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:59:24.693 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:59:24.694 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:59:24.694 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:59:24.694 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:59:24.694 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:59:24.694 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:59:24.694 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:24.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:24.695 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:59:24.695 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:59:24.695 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:59:24.695 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:59:24.695 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:59:24.695 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:59:24.695 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:59:24.696 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:59:24.696 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:59:24.696 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:59:24.696 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:24.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:24.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:59:24.696 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:59:24.697 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:59:24.697 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:59:24.697 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:59:24.697 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:59:24.697 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:59:24.697 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:59:24.697 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:59:24.698 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:59:24.698 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:24.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:24.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:59:24.698 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:59:24.698 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:59:24.699 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:59:24.699 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:59:24.699 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:59:24.699 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:59:24.699 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:59:24.699 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:59:24.699 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:59:24.699 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:24.700 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:24.700 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:59:24.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:24.700 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:59:24.700 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:59:24.700 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:59:24.700 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:24.701 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:59:24.702 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:59:29.703 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:59:29.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:59:29.704 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:59:29.745 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:59:29.745 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:59:30.118 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:59:30.118 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:59:30.118 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:59:30.118 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:59:30.119 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:59:30.119 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:59:30.119 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:59:30.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:30.119 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:59:30.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:59:30.119 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:59:30.120 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:59:30.120 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:59:30.120 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:59:30.120 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:59:30.120 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:59:30.120 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:59:30.120 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:59:30.121 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:59:30.121 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:30.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:30.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:59:30.121 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:59:30.121 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:59:30.121 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:59:30.122 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:59:30.122 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:59:30.122 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:59:30.122 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:59:30.122 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:59:30.122 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:59:30.122 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:30.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:30.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:59:30.123 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:59:30.123 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:59:30.123 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:59:30.123 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:59:30.123 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:59:30.123 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:59:30.124 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:59:30.124 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:59:30.124 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:59:30.124 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:30.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:30.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:59:30.124 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:59:30.125 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:59:30.125 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:59:30.125 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:59:30.125 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:59:30.125 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:59:30.125 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:59:30.125 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:59:30.125 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:59:30.126 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:30.126 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:30.126 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:59:30.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:30.126 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:59:30.126 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:59:30.126 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:59:30.127 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:30.127 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:59:30.128 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:59:35.129 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:59:35.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:59:35.131 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:59:35.171 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:59:35.171 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:59:35.541 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:59:35.541 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:59:35.541 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:59:35.542 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:59:35.542 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:59:35.542 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:59:35.542 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:59:35.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:35.542 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:59:35.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:59:35.543 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:59:35.543 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:59:35.543 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:59:35.543 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:59:35.543 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:59:35.543 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:59:35.544 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:59:35.544 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:59:35.544 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:59:35.544 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:35.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:35.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:59:35.545 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:59:35.545 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:59:35.545 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:59:35.545 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:59:35.545 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:59:35.545 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:59:35.546 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:59:35.546 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:59:35.546 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:59:35.546 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:35.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:35.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:59:35.546 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:59:35.547 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:59:35.547 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:59:35.547 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:59:35.547 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:59:35.547 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:59:35.547 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:59:35.547 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:59:35.547 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:59:35.548 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:35.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:35.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:59:35.548 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:59:35.548 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:59:35.548 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:59:35.548 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:59:35.549 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:59:35.549 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:59:35.549 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:59:35.549 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:59:35.549 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:59:35.549 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:35.549 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:35.550 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:59:35.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:35.550 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:59:35.550 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:59:35.550 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:59:35.550 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:35.550 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:59:35.551 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:59:40.553 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:59:40.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:59:40.554 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:59:40.594 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:59:40.595 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:59:40.956 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:59:40.956 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:59:40.956 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:59:40.957 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:59:40.957 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:59:40.957 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:59:40.957 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:59:40.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:40.957 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:59:40.957 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:59:40.958 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:59:40.958 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:59:40.958 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:59:40.958 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:59:40.958 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:59:40.958 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:59:40.959 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:59:40.959 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:59:40.959 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:59:40.959 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:40.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:40.959 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:59:40.959 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:59:40.959 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:59:40.960 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:59:40.960 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:59:40.960 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:59:40.960 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:59:40.960 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:59:40.960 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:59:40.960 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:59:40.961 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:40.961 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:40.961 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:59:40.961 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:59:40.961 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:59:40.961 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:59:40.961 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:59:40.962 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:59:40.962 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:59:40.962 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:59:40.962 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:59:40.962 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:59:40.962 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:40.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:40.962 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:59:40.963 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:59:40.963 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:59:40.963 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:59:40.963 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:59:40.963 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:59:40.963 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:59:40.963 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:59:40.964 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:59:40.964 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:59:40.964 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:40.964 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:40.964 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:59:40.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:40.964 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:59:40.964 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:59:40.965 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:59:40.965 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:40.965 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:59:40.966 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:59:45.967 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:59:45.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:59:45.969 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:59:46.011 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:59:46.011 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:59:46.381 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:59:46.381 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:59:46.381 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:59:46.382 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:59:46.382 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:59:46.382 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:59:46.382 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:59:46.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:46.382 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:59:46.382 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:59:46.383 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:59:46.383 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:59:46.383 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:59:46.383 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:59:46.383 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:59:46.383 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:59:46.383 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:59:46.384 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:59:46.384 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:59:46.384 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:46.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:46.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:59:46.384 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:59:46.384 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:59:46.385 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:59:46.385 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:59:46.385 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:59:46.385 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:59:46.385 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:59:46.386 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:59:46.386 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:59:46.386 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:46.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:46.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:59:46.386 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:59:46.386 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:59:46.387 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:59:46.387 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:59:46.387 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:59:46.387 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:59:46.387 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:59:46.387 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:59:46.387 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:59:46.387 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:46.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:46.388 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:59:46.388 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:59:46.388 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:59:46.388 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:59:46.388 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:59:46.388 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:59:46.389 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:59:46.389 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:59:46.389 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:59:46.389 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:59:46.389 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:46.389 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:46.389 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:59:46.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:46.390 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:59:46.390 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:59:46.390 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:59:46.390 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:46.390 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:59:46.391 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:59:51.392 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:59:51.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:59:51.393 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:59:51.435 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:59:51.436 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:59:51.801 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:59:51.801 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:59:51.802 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:59:51.802 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:59:51.802 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:59:51.802 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:59:51.802 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:59:51.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:51.803 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:59:51.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:59:51.803 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:59:51.804 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:59:51.804 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:59:51.804 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:59:51.804 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:59:51.804 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:59:51.804 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:59:51.805 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:59:51.805 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:59:51.805 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:51.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:51.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:59:51.805 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:59:51.806 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:59:51.806 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:59:51.806 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:59:51.806 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:59:51.807 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:59:51.807 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:59:51.807 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:59:51.807 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:59:51.807 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:51.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:51.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:59:51.808 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:59:51.808 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:59:51.808 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:59:51.808 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:59:51.808 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:59:51.808 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:59:51.808 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:59:51.809 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:59:51.809 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:59:51.809 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:51.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:51.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:59:51.809 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:59:51.809 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:59:51.809 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:59:51.810 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:59:51.810 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:59:51.810 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:59:51.810 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:59:51.810 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:59:51.810 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:59:51.811 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:51.811 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:51.811 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:59:51.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:51.811 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:59:51.811 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:59:51.811 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:59:51.811 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:51.812 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:59:51.813 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T18:59:56.814 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:59:56.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T18:59:56.815 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T18:59:56.855 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T18:59:56.855 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T18:59:57.224 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T18:59:57.224 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T18:59:57.224 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T18:59:57.224 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T18:59:57.224 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T18:59:57.225 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T18:59:57.225 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:59:57.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:57.225 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T18:59:57.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:59:57.225 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T18:59:57.225 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:59:57.225 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:59:57.226 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T18:59:57.226 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:59:57.226 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T18:59:57.226 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:59:57.226 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T18:59:57.226 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T18:59:57.226 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:57.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:57.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:59:57.227 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T18:59:57.227 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:59:57.227 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:59:57.227 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T18:59:57.227 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:59:57.228 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:59:57.228 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:59:57.228 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T18:59:57.228 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:59:57.228 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:57.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:57.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:59:57.229 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T18:59:57.229 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:59:57.229 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:59:57.229 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:59:57.229 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:59:57.229 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:59:57.229 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:59:57.229 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:59:57.230 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:59:57.230 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:57.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:57.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T18:59:57.230 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T18:59:57.230 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T18:59:57.230 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T18:59:57.231 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T18:59:57.231 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T18:59:57.231 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T18:59:57.231 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T18:59:57.231 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T18:59:57.231 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T18:59:57.231 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:57.232 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:57.232 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T18:59:57.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T18:59:57.232 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T18:59:57.232 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T18:59:57.232 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T18:59:57.232 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T18:59:57.232 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T18:59:57.233 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T19:00:02.234 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:00:02.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:00:02.236 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T19:00:02.275 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T19:00:02.275 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T19:00:02.636 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T19:00:02.636 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T19:00:02.636 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T19:00:02.636 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T19:00:02.637 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T19:00:02.637 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T19:00:02.637 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:00:02.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:02.637 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T19:00:02.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:00:02.637 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T19:00:02.638 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:00:02.638 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:00:02.638 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T19:00:02.638 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:00:02.638 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T19:00:02.638 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:00:02.638 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T19:00:02.639 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:00:02.639 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:02.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:02.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:00:02.639 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T19:00:02.639 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:00:02.639 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:00:02.640 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T19:00:02.640 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:00:02.640 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:00:02.640 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:00:02.640 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T19:00:02.640 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:00:02.640 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:02.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:02.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:00:02.641 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T19:00:02.641 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:00:02.641 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:00:02.641 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:00:02.641 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:00:02.642 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:00:02.642 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:00:02.642 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:00:02.642 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:00:02.642 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:02.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:02.642 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:00:02.642 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T19:00:02.643 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:00:02.643 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:00:02.643 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:00:02.643 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:00:02.643 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:00:02.643 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:00:02.644 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:00:02.644 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:00:02.644 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:02.644 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:02.644 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T19:00:02.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:02.644 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T19:00:02.645 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T19:00:02.645 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T19:00:02.645 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:02.645 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T19:00:02.646 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T19:00:07.648 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:00:07.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:00:07.649 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T19:00:07.690 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T19:00:07.690 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T19:00:08.055 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T19:00:08.056 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T19:00:08.056 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T19:00:08.056 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T19:00:08.056 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T19:00:08.056 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T19:00:08.056 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:00:08.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:08.057 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T19:00:08.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:00:08.057 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T19:00:08.057 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:00:08.057 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:00:08.057 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T19:00:08.058 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:00:08.058 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T19:00:08.058 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:00:08.058 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T19:00:08.058 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:00:08.058 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:08.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:08.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:00:08.059 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T19:00:08.059 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:00:08.059 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:00:08.059 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T19:00:08.059 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:00:08.059 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:00:08.059 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:00:08.060 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T19:00:08.060 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:00:08.060 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:08.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:08.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:00:08.060 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T19:00:08.060 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:00:08.061 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:00:08.061 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:00:08.061 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:00:08.061 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:00:08.061 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:00:08.061 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:00:08.061 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:00:08.062 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:08.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:08.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:00:08.062 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T19:00:08.062 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:00:08.062 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:00:08.062 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:00:08.063 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:00:08.063 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:00:08.063 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:00:08.063 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:00:08.063 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:00:08.063 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:08.063 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:08.063 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T19:00:08.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:08.064 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T19:00:08.064 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T19:00:08.064 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T19:00:08.064 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:08.064 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T19:00:08.065 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T19:00:13.066 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:00:13.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:00:13.068 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T19:00:13.108 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T19:00:13.108 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T19:00:13.478 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T19:00:13.479 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T19:00:13.479 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T19:00:13.479 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T19:00:13.479 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T19:00:13.481 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T19:00:13.481 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:00:13.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:13.482 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T19:00:13.482 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:00:13.482 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T19:00:13.482 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:00:13.482 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:00:13.483 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T19:00:13.483 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:00:13.483 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T19:00:13.485 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:00:13.485 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T19:00:13.485 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:00:13.485 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:13.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:13.485 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:00:13.486 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T19:00:13.486 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:00:13.486 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:00:13.486 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T19:00:13.486 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:00:13.486 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:00:13.486 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:00:13.487 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T19:00:13.488 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:00:13.488 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:13.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:13.488 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:00:13.488 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T19:00:13.489 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:00:13.490 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:00:13.490 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:00:13.490 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:00:13.490 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:00:13.490 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:00:13.491 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:00:13.491 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:00:13.491 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:13.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:13.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:00:13.492 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T19:00:13.492 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:00:13.492 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:00:13.492 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:00:13.492 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:00:13.493 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:00:13.493 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:00:13.493 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:00:13.493 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:00:13.493 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:13.494 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:13.494 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T19:00:13.494 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:13.494 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T19:00:13.494 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T19:00:13.495 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T19:00:13.495 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:13.495 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T19:00:13.496 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T19:00:18.498 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:00:18.498 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:00:18.499 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T19:00:18.536 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T19:00:18.537 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T19:00:18.902 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T19:00:19.475 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T19:00:19.475 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T19:00:19.476 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T19:00:19.476 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T19:00:19.476 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T19:00:19.476 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:00:19.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:19.477 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T19:00:19.477 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:00:19.478 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T19:00:19.478 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:00:19.478 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:00:19.478 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T19:00:19.479 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:00:19.479 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T19:00:19.479 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:00:19.479 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T19:00:19.480 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:00:19.480 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:19.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:19.481 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:00:19.481 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T19:00:19.482 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:00:19.482 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:00:19.482 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T19:00:19.482 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:00:19.482 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:00:19.483 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:00:19.483 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T19:00:19.483 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:00:19.483 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:19.484 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:19.485 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:00:19.485 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T19:00:19.485 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:00:19.485 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:00:19.485 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:00:19.485 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:00:19.485 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:00:19.486 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:00:19.486 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:00:19.486 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:00:19.486 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:19.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:19.486 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:00:19.486 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T19:00:19.487 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:00:19.487 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:00:19.487 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:00:19.487 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:00:19.487 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:00:19.487 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:00:19.487 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:00:19.488 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:00:19.488 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:19.488 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:19.488 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T19:00:19.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:19.488 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T19:00:19.488 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T19:00:19.489 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T19:00:19.489 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:19.489 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T19:00:19.490 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T19:00:24.492 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:00:24.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:00:24.494 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T19:00:24.534 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T19:00:24.534 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T19:00:24.907 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T19:00:24.907 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T19:00:24.907 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T19:00:24.908 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T19:00:24.908 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T19:00:24.908 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T19:00:24.908 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:00:24.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:24.908 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T19:00:24.908 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:00:24.909 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T19:00:24.909 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:00:24.909 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:00:24.909 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T19:00:24.909 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:00:24.909 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T19:00:24.910 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:00:24.910 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T19:00:24.910 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:00:24.910 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:24.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:24.911 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:00:24.911 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T19:00:24.911 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:00:24.911 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:00:24.911 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T19:00:24.911 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:00:24.912 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:00:24.912 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:00:24.912 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T19:00:24.913 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:00:24.913 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:24.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:24.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:00:24.913 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T19:00:24.913 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:00:24.913 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:00:24.914 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:00:24.914 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:00:24.914 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:00:24.914 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:00:24.914 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:00:24.914 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:00:24.915 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:24.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:24.915 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:00:24.915 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T19:00:24.915 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:00:24.915 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:00:24.915 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:00:24.916 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:00:24.916 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:00:24.916 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:00:24.916 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:00:24.916 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:00:24.916 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:24.917 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:24.917 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T19:00:24.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:24.917 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T19:00:24.917 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T19:00:24.917 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T19:00:24.917 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:24.918 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T19:00:24.919 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T19:00:29.920 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:00:29.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:00:29.921 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T19:00:29.962 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T19:00:29.963 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T19:00:30.333 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T19:00:30.333 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T19:00:30.334 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T19:00:30.334 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T19:00:30.334 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T19:00:30.334 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T19:00:30.335 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:00:30.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:30.335 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T19:00:30.336 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:00:30.336 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T19:00:30.336 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:00:30.337 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:00:30.337 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T19:00:30.337 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:00:30.337 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T19:00:30.338 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:00:30.338 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T19:00:30.338 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:00:30.339 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:30.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:30.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:00:30.340 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T19:00:30.340 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:00:30.340 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:00:30.340 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T19:00:30.341 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:00:30.341 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:00:30.341 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:00:30.342 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T19:00:30.342 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:00:30.342 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:30.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:30.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:00:30.343 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T19:00:30.343 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:00:30.344 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:00:30.344 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:00:30.344 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:00:30.345 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:00:30.345 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:00:30.345 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:00:30.345 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:00:30.346 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:30.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:30.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:00:30.347 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T19:00:30.347 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:00:30.347 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:00:30.348 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:00:30.348 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:00:30.348 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:00:30.348 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:00:30.349 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:00:30.349 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:00:30.349 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:30.350 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:30.350 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T19:00:30.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:30.351 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T19:00:30.351 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T19:00:30.351 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T19:00:30.351 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:30.352 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T19:00:30.353 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T19:00:35.355 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:00:35.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:00:35.356 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T19:00:35.397 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T19:00:35.398 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T19:00:35.765 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T19:00:35.765 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T19:00:35.765 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T19:00:35.766 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T19:00:35.766 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T19:00:35.766 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T19:00:35.766 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:00:35.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:35.766 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T19:00:35.767 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:00:35.767 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T19:00:35.767 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:00:35.767 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:00:35.767 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T19:00:35.767 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:00:35.767 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T19:00:35.768 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:00:35.768 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T19:00:35.768 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:00:35.768 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:35.768 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:35.768 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:00:35.769 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T19:00:35.769 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:00:35.769 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:00:35.769 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T19:00:35.769 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:00:35.769 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:00:35.769 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:00:35.770 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T19:00:35.770 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:00:35.770 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:35.770 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:35.770 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:00:35.770 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T19:00:35.770 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:00:35.771 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:00:35.771 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:00:35.771 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:00:35.771 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:00:35.771 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:00:35.771 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:00:35.771 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:00:35.772 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:35.772 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:35.772 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:00:35.772 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T19:00:35.772 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:00:35.772 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:00:35.772 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:00:35.773 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:00:35.773 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:00:35.773 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:00:35.773 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:00:35.773 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:00:35.773 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:35.773 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:35.774 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T19:00:35.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:35.774 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T19:00:35.774 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T19:00:35.774 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T19:00:35.774 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:35.775 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T19:00:35.776 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T19:00:40.777 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:00:40.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:00:40.779 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T19:00:40.817 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T19:00:40.817 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T19:00:41.289 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T19:00:41.289 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T19:00:41.290 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T19:00:41.290 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T19:00:41.290 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T19:00:41.290 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T19:00:41.290 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:00:41.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:41.291 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T19:00:41.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:00:41.291 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T19:00:41.291 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:00:41.291 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:00:41.291 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T19:00:41.291 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:00:41.292 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T19:00:41.292 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:00:41.292 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T19:00:41.292 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:00:41.292 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:41.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:41.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:00:41.293 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T19:00:41.293 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:00:41.293 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:00:41.293 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T19:00:41.293 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:00:41.293 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:00:41.293 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:00:41.293 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T19:00:41.294 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:00:41.294 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:41.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:41.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:00:41.294 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T19:00:41.294 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:00:41.294 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:00:41.295 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:00:41.295 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:00:41.295 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:00:41.295 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:00:41.295 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:00:41.295 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:00:41.295 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:41.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:41.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:00:41.296 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T19:00:41.296 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:00:41.296 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:00:41.296 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:00:41.296 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:00:41.296 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:00:41.297 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:00:41.297 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:00:41.297 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:00:41.297 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:41.297 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:41.297 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T19:00:41.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:41.298 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T19:00:41.298 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T19:00:41.298 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T19:00:41.298 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:41.298 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T19:00:41.299 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T19:00:46.301 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:00:46.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:00:46.302 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T19:00:46.344 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T19:00:46.344 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T19:00:46.713 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T19:00:46.713 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T19:00:46.714 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T19:00:46.714 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T19:00:46.714 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T19:00:46.714 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T19:00:46.714 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:00:46.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:46.714 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T19:00:46.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:00:46.715 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T19:00:46.715 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:00:46.715 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:00:46.715 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T19:00:46.715 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:00:46.715 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T19:00:46.716 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:00:46.716 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T19:00:46.716 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:00:46.716 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:46.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:46.716 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:00:46.716 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T19:00:46.717 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:00:46.717 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:00:46.717 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T19:00:46.717 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:00:46.717 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:00:46.717 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:00:46.717 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T19:00:46.718 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:00:46.718 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:46.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:46.718 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:00:46.718 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T19:00:46.718 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:00:46.718 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:00:46.719 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:00:46.719 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:00:46.719 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:00:46.719 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:00:46.719 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:00:46.719 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:00:46.719 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:46.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:46.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:00:46.720 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T19:00:46.720 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:00:46.720 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:00:46.720 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:00:46.720 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:00:46.720 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:00:46.721 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:00:46.721 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:00:46.721 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:00:46.721 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:46.721 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:46.721 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T19:00:46.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:46.722 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T19:00:46.722 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T19:00:46.722 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T19:00:46.722 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:46.722 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T19:00:46.723 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T19:00:51.725 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:00:51.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:00:51.726 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T19:00:51.767 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T19:00:51.768 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T19:00:52.132 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T19:00:52.132 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T19:00:52.132 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T19:00:52.133 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T19:00:52.133 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T19:00:52.133 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T19:00:52.133 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:00:52.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:52.133 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T19:00:52.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:00:52.134 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T19:00:52.134 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:00:52.134 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:00:52.134 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T19:00:52.134 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:00:52.134 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T19:00:52.134 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:00:52.135 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T19:00:52.135 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:00:52.135 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:52.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:52.135 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:00:52.135 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T19:00:52.135 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:00:52.136 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:00:52.136 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T19:00:52.136 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:00:52.136 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:00:52.136 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:00:52.136 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T19:00:52.136 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:00:52.137 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:52.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:52.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:00:52.137 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T19:00:52.137 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:00:52.137 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:00:52.137 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:00:52.137 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:00:52.138 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:00:52.138 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:00:52.138 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:00:52.138 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:00:52.138 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:52.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:52.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:00:52.139 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T19:00:52.139 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:00:52.139 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:00:52.139 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:00:52.139 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:00:52.139 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:00:52.139 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:00:52.140 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:00:52.140 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:00:52.140 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:52.140 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:52.140 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T19:00:52.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:52.140 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T19:00:52.141 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T19:00:52.141 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T19:00:52.141 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:52.141 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T19:00:52.142 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T19:00:57.143 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:00:57.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:00:57.144 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T19:00:57.186 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T19:00:57.187 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T19:00:57.551 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T19:00:57.551 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T19:00:57.551 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T19:00:57.551 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T19:00:57.552 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T19:00:57.552 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T19:00:57.552 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:00:57.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:57.552 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T19:00:57.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:00:57.553 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T19:00:57.553 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:00:57.553 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:00:57.553 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T19:00:57.553 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:00:57.554 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T19:00:57.554 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:00:57.554 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T19:00:57.554 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:00:57.554 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:57.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:57.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:00:57.555 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T19:00:57.556 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:00:57.556 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:00:57.556 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T19:00:57.556 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:00:57.556 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:00:57.556 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:00:57.556 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T19:00:57.557 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:00:57.557 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:57.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:57.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:00:57.557 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T19:00:57.557 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:00:57.557 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:00:57.558 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:00:57.558 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:00:57.558 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:00:57.558 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:00:57.558 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:00:57.558 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:00:57.559 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:57.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:57.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:00:57.559 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T19:00:57.559 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:00:57.559 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:00:57.559 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:00:57.560 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:00:57.560 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:00:57.560 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:00:57.560 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:00:57.560 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:00:57.560 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:57.561 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:57.561 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T19:00:57.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:00:57.561 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T19:00:57.561 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T19:00:57.561 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T19:00:57.561 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:00:57.562 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T19:00:57.563 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T19:01:02.564 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:01:02.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:01:02.565 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T19:01:02.609 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T19:01:02.610 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T19:01:02.966 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T19:01:02.967 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T19:01:02.968 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T19:01:02.968 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T19:01:02.968 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T19:01:02.970 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T19:01:02.970 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:01:02.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:02.970 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T19:01:02.971 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:01:02.971 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T19:01:02.971 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:01:02.971 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:01:02.971 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T19:01:02.971 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:01:02.971 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T19:01:02.972 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:01:02.972 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T19:01:02.972 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:01:02.972 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:02.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:02.972 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:01:02.972 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T19:01:02.973 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:01:02.973 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:01:02.973 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T19:01:02.973 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:01:02.973 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:01:02.973 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:01:02.973 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T19:01:02.974 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:01:02.974 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:02.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:02.974 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:01:02.974 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T19:01:02.974 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:01:02.974 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:01:02.974 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:01:02.975 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:01:02.975 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:01:02.975 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:01:02.975 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:01:02.975 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:01:02.975 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:02.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:02.976 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:01:02.976 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T19:01:02.976 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:01:02.976 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:01:02.976 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:01:02.976 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:01:02.976 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:01:02.977 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:01:02.977 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:01:02.977 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:01:02.977 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:02.977 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:02.977 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T19:01:02.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:02.977 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T19:01:02.978 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T19:01:02.978 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T19:01:02.978 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:02.978 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T19:01:02.979 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T19:01:07.980 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:01:07.981 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:01:07.982 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T19:01:08.022 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T19:01:08.022 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T19:01:08.488 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T19:01:08.488 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T19:01:08.488 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T19:01:08.489 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T19:01:08.489 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T19:01:08.489 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T19:01:08.489 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:01:08.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:08.489 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T19:01:08.490 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:01:08.490 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T19:01:08.490 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:01:08.490 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:01:08.490 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T19:01:08.490 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:01:08.491 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T19:01:08.491 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:01:08.491 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T19:01:08.491 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:01:08.491 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:08.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:08.492 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:01:08.492 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T19:01:08.492 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:01:08.492 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:01:08.492 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T19:01:08.492 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:01:08.492 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:01:08.493 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:01:08.493 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T19:01:08.493 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:01:08.493 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:08.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:08.493 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:01:08.494 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T19:01:08.494 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:01:08.494 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:01:08.494 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:01:08.494 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:01:08.494 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:01:08.495 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:01:08.495 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:01:08.495 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:01:08.495 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:08.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:08.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:01:08.496 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T19:01:08.496 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:01:08.496 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:01:08.496 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:01:08.496 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:01:08.496 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:01:08.496 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:01:08.497 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:01:08.497 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:01:08.497 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:08.497 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:08.497 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T19:01:08.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:08.498 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T19:01:08.498 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T19:01:08.498 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T19:01:08.498 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:08.498 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T19:01:08.499 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T19:01:13.501 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:01:13.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:01:13.724 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T19:01:13.767 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T19:01:13.767 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T19:01:14.139 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T19:01:14.139 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T19:01:14.139 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T19:01:14.139 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T19:01:14.140 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T19:01:14.140 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T19:01:14.140 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:01:14.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:14.140 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T19:01:14.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:01:14.140 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T19:01:14.141 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:01:14.141 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:01:14.141 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T19:01:14.141 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:01:14.141 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T19:01:14.141 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:01:14.141 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T19:01:14.142 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:01:14.142 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:14.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:14.142 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:01:14.142 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T19:01:14.142 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:01:14.142 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:01:14.143 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T19:01:14.143 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:01:14.143 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:01:14.143 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:01:14.143 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T19:01:14.144 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:01:14.145 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:14.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:14.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:01:14.145 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T19:01:14.145 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:01:14.146 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:01:14.146 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:01:14.146 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:01:14.146 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:01:14.146 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:01:14.146 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:01:14.147 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:01:14.147 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:14.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:14.147 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:01:14.147 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T19:01:14.147 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:01:14.148 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:01:14.148 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:01:14.148 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:01:14.148 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:01:14.148 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:01:14.148 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:01:14.148 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:01:14.149 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:14.149 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:14.149 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T19:01:14.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:14.149 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T19:01:14.149 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T19:01:14.149 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T19:01:14.150 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:14.150 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T19:01:14.151 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T19:01:19.152 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:01:19.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:01:19.155 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T19:01:19.199 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T19:01:19.200 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T19:01:19.569 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T19:01:19.569 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T19:01:19.569 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T19:01:19.570 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T19:01:19.570 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T19:01:19.570 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T19:01:19.570 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:01:19.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:19.570 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T19:01:19.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:01:19.571 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T19:01:19.571 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:01:19.571 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:01:19.571 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T19:01:19.571 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:01:19.571 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T19:01:19.571 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:01:19.572 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T19:01:19.572 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:01:19.572 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:19.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:19.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:01:19.572 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T19:01:19.572 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:01:19.573 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:01:19.573 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T19:01:19.573 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:01:19.573 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:01:19.573 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:01:19.573 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T19:01:19.573 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:01:19.574 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:19.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:19.574 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:01:19.574 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T19:01:19.574 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:01:19.574 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:01:19.574 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:01:19.575 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:01:19.575 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:01:19.575 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:01:19.575 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:01:19.575 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:01:19.575 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:19.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:19.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:01:19.576 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T19:01:19.576 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:01:19.576 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:01:19.576 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:01:19.576 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:01:19.576 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:01:19.576 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:01:19.577 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:01:19.577 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:01:19.577 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:19.577 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:19.577 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T19:01:19.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:19.577 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T19:01:19.578 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T19:01:19.578 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T19:01:19.578 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:19.578 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T19:01:19.579 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T19:01:24.581 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:01:24.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:01:24.582 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T19:01:24.622 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T19:01:24.622 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T19:01:25.094 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T19:01:25.095 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T19:01:25.095 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T19:01:25.095 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T19:01:25.095 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T19:01:25.095 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T19:01:25.096 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:01:25.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:25.096 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T19:01:25.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:01:25.096 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T19:01:25.096 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:01:25.097 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:01:25.097 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T19:01:25.097 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:01:25.097 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T19:01:25.097 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:01:25.097 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T19:01:25.097 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:01:25.097 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:25.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:25.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:01:25.098 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T19:01:25.098 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:01:25.098 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:01:25.098 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T19:01:25.098 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:01:25.099 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:01:25.099 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:01:25.099 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T19:01:25.099 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:01:25.099 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:25.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:25.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:01:25.100 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T19:01:25.100 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:01:25.100 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:01:25.100 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:01:25.100 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:01:25.100 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:01:25.101 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:01:25.101 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:01:25.101 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:01:25.101 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:25.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:25.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:01:25.101 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T19:01:25.102 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:01:25.102 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:01:25.102 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:01:25.102 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:01:25.102 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:01:25.102 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:01:25.102 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:01:25.103 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:01:25.103 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:25.103 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:25.103 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T19:01:25.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:25.103 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T19:01:25.103 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T19:01:25.103 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T19:01:25.104 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:25.104 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T19:01:25.105 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T19:01:30.106 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:01:30.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:01:30.108 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T19:01:30.149 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T19:01:30.149 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T19:01:30.515 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T19:01:30.515 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T19:01:30.515 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T19:01:30.515 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T19:01:30.515 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T19:01:30.516 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T19:01:30.516 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:01:30.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:30.516 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T19:01:30.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:01:30.516 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T19:01:30.516 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:01:30.517 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:01:30.517 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T19:01:30.517 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:01:30.517 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T19:01:30.517 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:01:30.517 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T19:01:30.517 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:01:30.518 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:30.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:30.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:01:30.518 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T19:01:30.518 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:01:30.518 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:01:30.518 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T19:01:30.519 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:01:30.519 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:01:30.519 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:01:30.519 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T19:01:30.519 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:01:30.519 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:30.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:30.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:01:30.520 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T19:01:30.520 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:01:30.520 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:01:30.521 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:01:30.521 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:01:30.521 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:01:30.521 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:01:30.521 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:01:30.521 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:01:30.521 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:30.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:30.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:01:30.522 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T19:01:30.522 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:01:30.522 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:01:30.522 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:01:30.522 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:01:30.523 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:01:30.523 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:01:30.523 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:01:30.523 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:01:30.523 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:30.523 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:30.523 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T19:01:30.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:30.524 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T19:01:30.524 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T19:01:30.524 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T19:01:30.525 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:30.525 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T19:01:30.526 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T19:01:35.527 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:01:35.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:01:35.529 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T19:01:35.569 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T19:01:35.570 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T19:01:35.936 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T19:01:35.936 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T19:01:35.937 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T19:01:35.937 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T19:01:35.937 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T19:01:35.937 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T19:01:35.937 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:01:35.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:35.938 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T19:01:35.938 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:01:35.938 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T19:01:35.938 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:01:35.938 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:01:35.938 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T19:01:35.938 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:01:35.939 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T19:01:35.939 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:01:35.939 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T19:01:35.939 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:01:35.939 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:35.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:35.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:01:35.939 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T19:01:35.940 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:01:35.940 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:01:35.940 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T19:01:35.940 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:01:35.940 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:01:35.940 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:01:35.940 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T19:01:35.941 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:01:35.941 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:35.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:35.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:01:35.941 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T19:01:35.941 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:01:35.941 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:01:35.942 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:01:35.942 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:01:35.942 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:01:35.942 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:01:35.942 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:01:35.942 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:01:35.942 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:35.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:35.943 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:01:35.943 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T19:01:35.943 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:01:35.943 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:01:35.943 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:01:35.943 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:01:35.943 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:01:35.944 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:01:35.944 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:01:35.944 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:01:35.944 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:35.944 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:35.944 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T19:01:35.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:35.945 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T19:01:35.945 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T19:01:35.945 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T19:01:35.945 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:35.945 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T19:01:35.946 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T19:01:40.948 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:01:40.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:01:40.949 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T19:01:40.991 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T19:01:40.991 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T19:01:41.356 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T19:01:41.357 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T19:01:41.357 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T19:01:41.357 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T19:01:41.358 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T19:01:41.358 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T19:01:41.358 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:01:41.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:41.359 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T19:01:41.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:01:41.360 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T19:01:41.360 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:01:41.360 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:01:41.360 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T19:01:41.361 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:01:41.361 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T19:01:41.361 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:01:41.362 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T19:01:41.362 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:01:41.362 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:41.362 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:41.363 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:01:41.363 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T19:01:41.363 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:01:41.364 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:01:41.364 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T19:01:41.365 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:01:41.365 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:01:41.365 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:01:41.366 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T19:01:41.366 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:01:41.366 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:41.366 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:41.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:01:41.367 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T19:01:41.367 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:01:41.368 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:01:41.368 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:01:41.368 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:01:41.368 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:01:41.369 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:01:41.369 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:01:41.369 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:01:41.369 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:41.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:41.370 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:01:41.370 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T19:01:41.371 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:01:41.371 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:01:41.371 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:01:41.371 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:01:41.372 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:01:41.372 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:01:41.372 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:01:41.373 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:01:41.373 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:41.373 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:41.373 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T19:01:41.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:41.374 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T19:01:41.374 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T19:01:41.375 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T19:01:41.375 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:41.375 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T19:01:41.377 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T19:01:46.378 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:01:46.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:01:46.379 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T19:01:46.422 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T19:01:46.422 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T19:01:46.782 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T19:01:46.783 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T19:01:46.783 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T19:01:46.783 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T19:01:46.783 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T19:01:46.783 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T19:01:46.783 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:01:46.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:46.784 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T19:01:46.784 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:01:46.784 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T19:01:46.784 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:01:46.784 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:01:46.784 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T19:01:46.785 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:01:46.785 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T19:01:46.785 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:01:46.785 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T19:01:46.785 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:01:46.785 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:46.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:46.786 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:01:46.786 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T19:01:46.786 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:01:46.786 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:01:46.786 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T19:01:46.786 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:01:46.787 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:01:46.787 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:01:46.787 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T19:01:46.787 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:01:46.787 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:46.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:46.787 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:01:46.787 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T19:01:46.788 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:01:46.788 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:01:46.788 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:01:46.788 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:01:46.788 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:01:46.788 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:01:46.788 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:01:46.789 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:01:46.789 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:46.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:46.789 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:01:46.789 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T19:01:46.789 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:01:46.789 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:01:46.790 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:01:46.790 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:01:46.790 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:01:46.790 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:01:46.790 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:01:46.790 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:01:46.790 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:46.791 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:46.791 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T19:01:46.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:46.791 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T19:01:46.791 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T19:01:46.791 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T19:01:46.791 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:46.792 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T19:01:46.793 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T19:01:51.794 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:01:51.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:01:51.796 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T19:01:51.837 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T19:01:51.837 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T19:01:52.204 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T19:01:52.205 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T19:01:52.205 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T19:01:52.206 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T19:01:52.206 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T19:01:52.206 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T19:01:52.207 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:01:52.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:52.207 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T19:01:52.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:01:52.208 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T19:01:52.208 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:01:52.208 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:01:52.209 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T19:01:52.209 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:01:52.209 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T19:01:52.210 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:01:52.210 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T19:01:52.210 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:01:52.211 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:52.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:52.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:01:52.212 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T19:01:52.213 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:01:52.213 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:01:52.213 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T19:01:52.213 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:01:52.214 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:01:52.214 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:01:52.214 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T19:01:52.215 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:01:52.215 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:52.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:52.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:01:52.216 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T19:01:52.216 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:01:52.217 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:01:52.217 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:01:52.217 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:01:52.219 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:01:52.219 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:01:52.219 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:01:52.220 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:01:52.220 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:52.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:52.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:01:52.221 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T19:01:52.221 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:01:52.221 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:01:52.222 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:01:52.222 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:01:52.222 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:01:52.223 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:01:52.223 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:01:52.223 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:01:52.224 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:52.224 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:52.224 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T19:01:52.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:52.225 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T19:01:52.225 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T19:01:52.225 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T19:01:52.226 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:52.226 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T19:01:52.227 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T19:01:57.229 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:01:57.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:01:57.231 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T19:01:57.270 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T19:01:57.271 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T19:01:57.637 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T19:01:57.638 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T19:01:57.638 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T19:01:57.638 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T19:01:57.638 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T19:01:57.638 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T19:01:57.638 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:01:57.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:57.640 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T19:01:57.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:01:57.640 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T19:01:57.640 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:01:57.640 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:01:57.640 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T19:01:57.641 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:01:57.641 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T19:01:57.641 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:01:57.641 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T19:01:57.641 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:01:57.642 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:57.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:57.642 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:01:57.642 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T19:01:57.642 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:01:57.642 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:01:57.643 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T19:01:57.643 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:01:57.643 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:01:57.643 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:01:57.643 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T19:01:57.643 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:01:57.643 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:57.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:57.644 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:01:57.644 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T19:01:57.644 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:01:57.644 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:01:57.644 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:01:57.644 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:01:57.645 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:01:57.645 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:01:57.645 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:01:57.645 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:01:57.646 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:57.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:57.646 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:01:57.646 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T19:01:57.646 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:01:57.646 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:01:57.647 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:01:57.647 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:01:57.647 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:01:57.647 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:01:57.647 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:01:57.648 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:01:57.648 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:57.648 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:57.648 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T19:01:57.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:01:57.648 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T19:01:57.649 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T19:01:57.649 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T19:01:57.649 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:01:57.649 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T19:01:57.650 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T19:02:02.651 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:02:02.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:02:02.653 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T19:02:02.693 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T19:02:02.694 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T19:02:03.062 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T19:02:03.063 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T19:02:03.063 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T19:02:03.063 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T19:02:03.063 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T19:02:03.064 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T19:02:03.064 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:02:03.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:03.064 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T19:02:03.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:02:03.064 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T19:02:03.064 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:02:03.065 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:02:03.065 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T19:02:03.065 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:02:03.065 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T19:02:03.065 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:02:03.065 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T19:02:03.065 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:02:03.066 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:03.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:03.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:02:03.066 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T19:02:03.066 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:02:03.066 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:02:03.066 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T19:02:03.067 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:02:03.067 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:02:03.067 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:02:03.067 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T19:02:03.068 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:02:03.068 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:03.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:03.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:02:03.068 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T19:02:03.068 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:02:03.068 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:02:03.068 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:02:03.069 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:02:03.069 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:02:03.069 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:02:03.069 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:02:03.069 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:02:03.069 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:03.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:03.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:02:03.070 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T19:02:03.070 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:02:03.070 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:02:03.070 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:02:03.070 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:02:03.070 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:02:03.071 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:02:03.071 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:02:03.071 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:02:03.071 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:03.071 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:03.071 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T19:02:03.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:03.072 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T19:02:03.072 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T19:02:03.072 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T19:02:03.072 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:03.072 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T19:02:03.073 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T19:02:08.074 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:02:08.075 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:02:08.075 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T19:02:08.114 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T19:02:08.115 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T19:02:08.484 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T19:02:08.485 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T19:02:08.485 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T19:02:08.485 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T19:02:08.485 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T19:02:08.485 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T19:02:08.486 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:02:08.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:08.486 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T19:02:08.486 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:02:08.486 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T19:02:08.487 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:02:08.487 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:02:08.487 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T19:02:08.487 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:02:08.488 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T19:02:08.488 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:02:08.488 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T19:02:08.488 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:02:08.488 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:08.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:08.488 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:02:08.489 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T19:02:08.489 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:02:08.489 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:02:08.489 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T19:02:08.489 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:02:08.489 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:02:08.489 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:02:08.490 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T19:02:08.490 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:02:08.490 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:08.490 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:08.490 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:02:08.490 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T19:02:08.491 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:02:08.491 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:02:08.491 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:02:08.491 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:02:08.491 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:02:08.491 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:02:08.491 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:02:08.492 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:02:08.492 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:08.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:08.492 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:02:08.492 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T19:02:08.492 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:02:08.492 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:02:08.493 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:02:08.493 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:02:08.493 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:02:08.493 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:02:08.493 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:02:08.493 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:02:08.493 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:08.494 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:08.494 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T19:02:08.494 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:08.494 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T19:02:08.494 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T19:02:08.494 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T19:02:08.495 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:08.495 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T19:02:08.496 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T19:02:13.497 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:02:13.498 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:02:13.498 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T19:02:13.538 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T19:02:13.538 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T19:02:13.917 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T19:02:13.917 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T19:02:13.918 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T19:02:13.918 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T19:02:13.918 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T19:02:13.918 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T19:02:13.918 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:02:13.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:13.918 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T19:02:13.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:02:13.919 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T19:02:13.919 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:02:13.919 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:02:13.919 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T19:02:13.919 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:02:13.919 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T19:02:13.920 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:02:13.920 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T19:02:13.920 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:02:13.920 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:13.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:13.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:02:13.920 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T19:02:13.921 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:02:13.921 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:02:13.921 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T19:02:13.921 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:02:13.921 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:02:13.921 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:02:13.921 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T19:02:13.922 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:02:13.922 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:13.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:13.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:02:13.922 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T19:02:13.922 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:02:13.922 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:02:13.922 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:02:13.923 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:02:13.923 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:02:13.923 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:02:13.923 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:02:13.923 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:02:13.923 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:13.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:13.924 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:02:13.924 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T19:02:13.924 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:02:13.924 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:02:13.924 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:02:13.924 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:02:13.925 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:02:13.925 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:02:13.925 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:02:13.925 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:02:13.925 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:13.925 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:13.925 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T19:02:13.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:13.926 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T19:02:13.926 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T19:02:13.926 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T19:02:13.926 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:13.926 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T19:02:13.927 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T19:02:18.929 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:02:18.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:02:18.930 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T19:02:18.970 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T19:02:18.971 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T19:02:19.345 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T19:02:19.346 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T19:02:19.346 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T19:02:19.346 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T19:02:19.346 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T19:02:19.346 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T19:02:19.347 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:02:19.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:19.347 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T19:02:19.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:02:19.347 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T19:02:19.347 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:02:19.347 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:02:19.348 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T19:02:19.348 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:02:19.348 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T19:02:19.348 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:02:19.348 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T19:02:19.348 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:02:19.348 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:19.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:19.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:02:19.349 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T19:02:19.349 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:02:19.349 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:02:19.349 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T19:02:19.349 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:02:19.350 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:02:19.350 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:02:19.350 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T19:02:19.350 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:02:19.350 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:19.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:19.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:02:19.353 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T19:02:19.353 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:02:19.353 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:02:19.354 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:02:19.354 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:02:19.354 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:02:19.354 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:02:19.354 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:02:19.355 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:02:19.355 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:19.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:19.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:02:19.355 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T19:02:19.355 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:02:19.356 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:02:19.356 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:02:19.356 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:02:19.356 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:02:19.356 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:02:19.356 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:02:19.357 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:02:19.357 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:19.357 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:19.357 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T19:02:19.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:19.357 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T19:02:19.358 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T19:02:19.358 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T19:02:19.358 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:19.358 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T19:02:19.359 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T19:02:24.360 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:02:24.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:02:24.362 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T19:02:24.402 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T19:02:24.402 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T19:02:24.777 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T19:02:24.778 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T19:02:24.778 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T19:02:24.778 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T19:02:24.778 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T19:02:24.778 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T19:02:24.778 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:02:24.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:24.779 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T19:02:24.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:02:24.779 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T19:02:24.779 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:02:24.779 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:02:24.780 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T19:02:24.780 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:02:24.780 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T19:02:24.780 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:02:24.780 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T19:02:24.780 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:02:24.781 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:24.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:24.781 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:02:24.781 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T19:02:24.781 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:02:24.781 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:02:24.782 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T19:02:24.782 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:02:24.782 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:02:24.782 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:02:24.782 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T19:02:24.782 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:02:24.783 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:24.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:24.783 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:02:24.783 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T19:02:24.783 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:02:24.783 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:02:24.783 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:02:24.784 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:02:24.784 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:02:24.784 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:02:24.784 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:02:24.784 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:02:24.784 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:24.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:24.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:02:24.785 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T19:02:24.785 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:02:24.785 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:02:24.785 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:02:24.786 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:02:24.786 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:02:24.786 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:02:24.786 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:02:24.786 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:02:24.786 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:24.787 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:24.787 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T19:02:24.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:24.787 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T19:02:24.787 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T19:02:24.787 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T19:02:24.787 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:24.788 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T19:02:24.788 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T19:02:29.790 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:02:29.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:02:29.791 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T19:02:29.830 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T19:02:29.831 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T19:02:30.194 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T19:02:30.195 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T19:02:30.195 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T19:02:30.195 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T19:02:30.195 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T19:02:30.195 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T19:02:30.196 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:02:30.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:30.196 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T19:02:30.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:02:30.196 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T19:02:30.196 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:02:30.197 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:02:30.197 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T19:02:30.197 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:02:30.197 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T19:02:30.197 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:02:30.197 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T19:02:30.197 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:02:30.198 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:30.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:30.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:02:30.198 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T19:02:30.198 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:02:30.199 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:02:30.199 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T19:02:30.199 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:02:30.199 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:02:30.199 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:02:30.199 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T19:02:30.200 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:02:30.200 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:30.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:30.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:02:30.200 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T19:02:30.200 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:02:30.201 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:02:30.201 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:02:30.201 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:02:30.201 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:02:30.201 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:02:30.201 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:02:30.201 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:02:30.202 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:30.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:30.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:02:30.202 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T19:02:30.202 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:02:30.202 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:02:30.203 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:02:30.203 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:02:30.203 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:02:30.203 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:02:30.203 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:02:30.203 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:02:30.203 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:30.204 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:30.204 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T19:02:30.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:30.204 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T19:02:30.204 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T19:02:30.204 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T19:02:30.205 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:30.205 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T19:02:30.206 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T19:02:35.207 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:02:35.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:02:35.208 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T19:02:35.249 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T19:02:35.250 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T19:02:35.615 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T19:02:35.615 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T19:02:35.616 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T19:02:35.616 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T19:02:35.616 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T19:02:35.616 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T19:02:35.616 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:02:35.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:35.617 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T19:02:35.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:02:35.617 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T19:02:35.617 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:02:35.617 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:02:35.617 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T19:02:35.617 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:02:35.618 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T19:02:35.618 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:02:35.618 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T19:02:35.618 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:02:35.618 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:35.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:35.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:02:35.619 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T19:02:35.619 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:02:35.619 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:02:35.619 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T19:02:35.619 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:02:35.619 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:02:35.619 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:02:35.620 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T19:02:35.620 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:02:35.620 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:35.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:35.620 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:02:35.620 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T19:02:35.620 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:02:35.620 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:02:35.621 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:02:35.621 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:02:35.621 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:02:35.621 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:02:35.621 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:02:35.621 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:02:35.621 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:35.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:35.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:02:35.622 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T19:02:35.622 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:02:35.622 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:02:35.622 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:02:35.622 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:02:35.623 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:02:35.623 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:02:35.623 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:02:35.623 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:02:35.623 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:35.623 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:35.623 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T19:02:35.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:35.624 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T19:02:35.624 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T19:02:35.624 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T19:02:35.624 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:35.624 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T19:02:35.625 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T19:02:40.627 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:02:40.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:02:40.628 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T19:02:40.666 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T19:02:40.667 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T19:02:41.026 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T19:02:41.027 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T19:02:41.027 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T19:02:41.027 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T19:02:41.027 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T19:02:41.027 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T19:02:41.027 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:02:41.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:41.028 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T19:02:41.028 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:02:41.028 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T19:02:41.028 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:02:41.028 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:02:41.028 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T19:02:41.029 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:02:41.029 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T19:02:41.029 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:02:41.029 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T19:02:41.029 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:02:41.029 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:41.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:41.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:02:41.030 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T19:02:41.030 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:02:41.030 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:02:41.030 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T19:02:41.030 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:02:41.030 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:02:41.031 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:02:41.031 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T19:02:41.031 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:02:41.031 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:41.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:41.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:02:41.031 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T19:02:41.032 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:02:41.032 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:02:41.032 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:02:41.032 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:02:41.032 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:02:41.032 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:02:41.032 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:02:41.032 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:02:41.033 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:41.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:41.033 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:02:41.033 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T19:02:41.033 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:02:41.033 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:02:41.033 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:02:41.034 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:02:41.034 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:02:41.034 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:02:41.034 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:02:41.034 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:02:41.034 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:41.034 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:41.035 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T19:02:41.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:41.035 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T19:02:41.035 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T19:02:41.035 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T19:02:41.035 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:41.035 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T19:02:41.036 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T19:02:46.037 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:02:46.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:02:46.039 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T19:02:46.080 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T19:02:46.081 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T19:02:46.444 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T19:02:46.444 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T19:02:46.445 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T19:02:46.445 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T19:02:46.446 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T19:02:46.446 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T19:02:46.446 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:02:46.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:46.446 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T19:02:46.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:02:46.447 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T19:02:46.447 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:02:46.447 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:02:46.447 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T19:02:46.447 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:02:46.448 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T19:02:46.448 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:02:46.448 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T19:02:46.448 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:02:46.448 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:46.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:46.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:02:46.449 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T19:02:46.449 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:02:46.450 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:02:46.450 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T19:02:46.450 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:02:46.450 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:02:46.450 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:02:46.451 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T19:02:46.451 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:02:46.451 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:46.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:46.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:02:46.451 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T19:02:46.452 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:02:46.452 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:02:46.452 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:02:46.452 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:02:46.452 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:02:46.452 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:02:46.453 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:02:46.453 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:02:46.453 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:46.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:46.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:02:46.454 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T19:02:46.454 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:02:46.454 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:02:46.454 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:02:46.454 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:02:46.455 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:02:46.455 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:02:46.455 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:02:46.455 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:02:46.455 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:46.455 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:46.456 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T19:02:46.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:46.456 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T19:02:46.456 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T19:02:46.456 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T19:02:46.456 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:46.456 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T19:02:46.457 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T19:02:51.459 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:02:51.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:02:51.460 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T19:02:51.502 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T19:02:51.503 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T19:02:51.771 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T19:02:51.772 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T19:02:51.772 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T19:02:51.772 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T19:02:51.772 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T19:02:51.772 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T19:02:51.772 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:02:51.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:51.773 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T19:02:51.773 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:02:51.773 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T19:02:51.773 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:02:51.773 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:02:51.774 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T19:02:51.774 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:02:51.774 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T19:02:51.774 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:02:51.774 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T19:02:51.775 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:02:51.775 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:51.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:51.775 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:02:51.775 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T19:02:51.775 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:02:51.775 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:02:51.776 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T19:02:51.776 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:02:51.776 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:02:51.776 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:02:51.776 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T19:02:51.776 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:02:51.776 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:51.776 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:51.777 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:02:51.777 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T19:02:51.777 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:02:51.777 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:02:51.777 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:02:51.777 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:02:51.777 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:02:51.778 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:02:51.778 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:02:51.778 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:02:51.778 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:51.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:51.778 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:02:51.778 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T19:02:51.779 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:02:51.779 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:02:51.779 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:02:51.779 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:02:51.779 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:02:51.779 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:02:51.780 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:02:51.780 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:02:51.780 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:51.781 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:51.781 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T19:02:51.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:51.781 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T19:02:51.782 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T19:02:51.782 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T19:02:51.782 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:51.783 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T19:02:51.784 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T19:02:56.785 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:02:56.786 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:02:56.787 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T19:02:56.828 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T19:02:56.829 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T19:02:57.192 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T19:02:57.193 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T19:02:57.193 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T19:02:57.193 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T19:02:57.193 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T19:02:57.193 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T19:02:57.193 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:02:57.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:57.194 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T19:02:57.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:02:57.194 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T19:02:57.194 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:02:57.194 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:02:57.194 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T19:02:57.195 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:02:57.195 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T19:02:57.195 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:02:57.195 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T19:02:57.195 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:02:57.195 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:57.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:57.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:02:57.196 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T19:02:57.196 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:02:57.196 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:02:57.196 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T19:02:57.196 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:02:57.196 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:02:57.196 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:02:57.197 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T19:02:57.197 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:02:57.197 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:57.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:57.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:02:57.197 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T19:02:57.197 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:02:57.198 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:02:57.198 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:02:57.198 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:02:57.198 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:02:57.198 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:02:57.198 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:02:57.198 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:02:57.199 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:57.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:57.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:02:57.199 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T19:02:57.199 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:02:57.199 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:02:57.199 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:02:57.200 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:02:57.200 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:02:57.200 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:02:57.200 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:02:57.200 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:02:57.201 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:57.201 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:57.201 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T19:02:57.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:02:57.201 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T19:02:57.202 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T19:02:57.202 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T19:02:57.202 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:02:57.202 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T19:02:57.203 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T19:03:02.204 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:03:02.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:03:02.205 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T19:03:02.247 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T19:03:02.248 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T19:03:02.612 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T19:03:02.612 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T19:03:02.612 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T19:03:02.613 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T19:03:02.613 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T19:03:02.613 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T19:03:02.613 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:03:02.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:03:02.613 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T19:03:02.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:03:02.614 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T19:03:02.614 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:03:02.614 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:03:02.615 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T19:03:02.615 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:03:02.615 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T19:03:02.615 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:03:02.615 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T19:03:02.615 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:03:02.616 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:03:02.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:03:02.616 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:03:02.616 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T19:03:02.616 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:03:02.616 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:03:02.616 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T19:03:02.617 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:03:02.617 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:03:02.617 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:03:02.617 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T19:03:02.617 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:03:02.617 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:03:02.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:03:02.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:03:02.618 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T19:03:02.618 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:03:02.618 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:03:02.618 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:03:02.618 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:03:02.619 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:03:02.619 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:03:02.619 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:03:02.619 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:03:02.619 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:03:02.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:03:02.620 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:03:02.620 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T19:03:02.620 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:03:02.620 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:03:02.620 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:03:02.620 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:03:02.620 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:03:02.621 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:03:02.621 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:03:02.621 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:03:02.621 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:03:02.621 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:03:02.621 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T19:03:02.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:03:02.622 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T19:03:02.622 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T19:03:02.622 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T19:03:02.622 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:03:02.622 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T19:03:02.623 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T19:03:07.625 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:03:07.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:03:07.626 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T19:03:07.659 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T19:03:07.660 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T19:03:08.021 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T19:03:08.022 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T19:03:08.022 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T19:03:08.022 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T19:03:08.022 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T19:03:08.022 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T19:03:08.022 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:03:08.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:03:08.023 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T19:03:08.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:03:08.023 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T19:03:08.023 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:03:08.024 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:03:08.024 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T19:03:08.024 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:03:08.024 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T19:03:08.024 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:03:08.025 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T19:03:08.025 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:03:08.025 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:03:08.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:03:08.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:03:08.026 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T19:03:08.026 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:03:08.026 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:03:08.027 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T19:03:08.027 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:03:08.027 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:03:08.027 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:03:08.028 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T19:03:08.028 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:03:08.028 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:03:08.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:03:08.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:03:08.029 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T19:03:08.029 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:03:08.029 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:03:08.030 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:03:08.030 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:03:08.030 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:03:08.031 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:03:08.031 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:03:08.031 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:03:08.031 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:03:08.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:03:08.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:03:08.032 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T19:03:08.033 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:03:08.033 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:03:08.033 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:03:08.033 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:03:08.034 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:03:08.034 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:03:08.034 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:03:08.035 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:03:08.035 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:03:08.035 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:03:08.035 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T19:03:08.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:03:08.036 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T19:03:08.036 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T19:03:08.037 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T19:03:08.037 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:03:08.037 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T19:03:08.039 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T19:03:13.041 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:03:13.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:03:13.042 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T19:03:13.083 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T19:03:13.084 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T19:03:13.449 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T19:03:13.450 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T19:03:13.450 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T19:03:13.450 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T19:03:13.450 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T19:03:13.450 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T19:03:13.451 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:03:13.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:03:13.451 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T19:03:13.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:03:13.451 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T19:03:13.451 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:03:13.451 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:03:13.452 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T19:03:13.452 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:03:13.452 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T19:03:13.452 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:03:13.452 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T19:03:13.452 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:03:13.452 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:03:13.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:03:13.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:03:13.453 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T19:03:13.453 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:03:13.453 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:03:13.453 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T19:03:13.453 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:03:13.454 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:03:13.454 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:03:13.454 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T19:03:13.454 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:03:13.454 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:03:13.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:03:13.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:03:13.455 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T19:03:13.455 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:03:13.455 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:03:13.456 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:03:13.456 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:03:13.456 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:03:13.456 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:03:13.456 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:03:13.456 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:03:13.456 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:03:13.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:03:13.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:03:13.457 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T19:03:13.457 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:03:13.457 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:03:13.457 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:03:13.457 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:03:13.458 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:03:13.458 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:03:13.458 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:03:13.458 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:03:13.458 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:03:13.458 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:03:13.458 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T19:03:13.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:03:13.459 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T19:03:13.459 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T19:03:13.459 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T19:03:13.459 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:03:13.459 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T19:03:13.460 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T19:03:18.461 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:03:18.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:03:18.464 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T19:03:18.503 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T19:03:18.503 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T19:03:18.775 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T19:03:18.775 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T19:03:18.775 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T19:03:18.776 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T19:03:18.776 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T19:03:18.776 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T19:03:18.776 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:03:18.776 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:03:18.777 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T19:03:18.777 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:03:18.777 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T19:03:18.777 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:03:18.777 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:03:18.778 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T19:03:18.778 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:03:18.778 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T19:03:18.778 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:03:18.778 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T19:03:18.778 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:03:18.779 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:03:18.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:03:18.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:03:18.779 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T19:03:18.779 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:03:18.780 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:03:18.780 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T19:03:18.780 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:03:18.780 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:03:18.781 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:03:18.782 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T19:03:18.783 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:03:18.783 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:03:18.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:03:18.784 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:03:18.786 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T19:03:18.786 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:03:18.786 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:03:18.787 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:03:18.787 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:03:18.787 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:03:18.787 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:03:18.787 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:03:18.788 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:03:18.788 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:03:18.788 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:03:18.788 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:03:18.788 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T19:03:18.789 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:03:18.789 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:03:18.789 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:03:18.789 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:03:18.789 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:03:18.790 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:03:18.790 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:03:18.790 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:03:18.790 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:03:18.791 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:03:18.791 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T19:03:18.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:03:18.791 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T19:03:18.791 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T19:03:18.792 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T19:03:18.792 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:03:18.792 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T19:03:18.793 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T19:03:23.795 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:03:23.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:03:23.796 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T19:03:23.839 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T19:03:23.839 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T19:03:24.208 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2020-11-09T19:03:24.209 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_status": { 2020-11-09T19:03:24.209 INFO:teuthology.orchestra.run.smithi078.stdout: "info": { 2020-11-09T19:03:24.209 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "sync", 2020-11-09T19:03:24.209 INFO:teuthology.orchestra.run.smithi078.stdout: "num_shards": 4, 2020-11-09T19:03:24.209 INFO:teuthology.orchestra.run.smithi078.stdout: "period": "f6258aba-300a-4d5d-ad93-a7c198a0e0a1", 2020-11-09T19:03:24.209 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:03:24.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:03:24.210 INFO:teuthology.orchestra.run.smithi078.stdout: "markers": [ 2020-11-09T19:03:24.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:03:24.210 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 0, 2020-11-09T19:03:24.210 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:03:24.210 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:03:24.210 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604946337.732873_438.1", 2020-11-09T19:03:24.211 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:03:24.211 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 1, 2020-11-09T19:03:24.211 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:03:24.211 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:25:37.732873Z", 2020-11-09T19:03:24.211 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 4 2020-11-09T19:03:24.211 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:03:24.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:03:24.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:03:24.212 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 1, 2020-11-09T19:03:24.212 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:03:24.212 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:03:24.212 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "1_1604945582.432939_95.1", 2020-11-09T19:03:24.212 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:03:24.212 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:03:24.213 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:03:24.213 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "2020-11-09T18:13:02.432939Z", 2020-11-09T19:03:24.213 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:03:24.213 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:03:24.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:03:24.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:03:24.213 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 2, 2020-11-09T19:03:24.213 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:03:24.214 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:03:24.214 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:03:24.214 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:03:24.214 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:03:24.214 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:03:24.214 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:03:24.214 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:03:24.215 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:03:24.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:03:24.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2020-11-09T19:03:24.215 INFO:teuthology.orchestra.run.smithi078.stdout: "key": 3, 2020-11-09T19:03:24.215 INFO:teuthology.orchestra.run.smithi078.stdout: "val": { 2020-11-09T19:03:24.215 INFO:teuthology.orchestra.run.smithi078.stdout: "state": 1, 2020-11-09T19:03:24.215 INFO:teuthology.orchestra.run.smithi078.stdout: "marker": "", 2020-11-09T19:03:24.216 INFO:teuthology.orchestra.run.smithi078.stdout: "next_step_marker": "", 2020-11-09T19:03:24.216 INFO:teuthology.orchestra.run.smithi078.stdout: "total_entries": 0, 2020-11-09T19:03:24.216 INFO:teuthology.orchestra.run.smithi078.stdout: "pos": 0, 2020-11-09T19:03:24.216 INFO:teuthology.orchestra.run.smithi078.stdout: "timestamp": "0.000000", 2020-11-09T19:03:24.216 INFO:teuthology.orchestra.run.smithi078.stdout: "realm_epoch": 2 2020-11-09T19:03:24.216 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:03:24.216 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:03:24.216 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2020-11-09T19:03:24.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2020-11-09T19:03:24.217 INFO:teuthology.orchestra.run.smithi078.stdout: "full_sync": { 2020-11-09T19:03:24.217 INFO:teuthology.orchestra.run.smithi078.stdout: "total": 1, 2020-11-09T19:03:24.217 INFO:teuthology.orchestra.run.smithi078.stdout: "complete": 1 2020-11-09T19:03:24.217 INFO:teuthology.orchestra.run.smithi078.stdout: } 2020-11-09T19:03:24.217 INFO:teuthology.orchestra.run.smithi078.stdout:} 2020-11-09T19:03:24.218 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1604946340.774139_451.1 target=1_1604946337.732873_438.1 2020-11-09T19:03:29.219 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:03:29.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T19:03:29.221 INFO:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T19:03:29.262 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setuser ceph since I am not root 2020-11-09T19:03:29.262 INFO:teuthology.orchestra.run.smithi078.stderr:ignoring --setgroup ceph since I am not root 2020-11-10T05:53:58.407 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2020-11-10T05:53:59.082 DEBUG:teuthology.task.console_log:Killing console logger for smithi051 2020-11-10T05:53:59.083 DEBUG:teuthology.task.console_log:Killing console logger for smithi078 2020-11-10T05:53:59.084 DEBUG:teuthology.task.console_log:Killing console logger for smithi051 2020-11-10T05:53:59.084 DEBUG:teuthology.task.console_log:Killing console logger for smithi078 2020-11-10T05:53:59.084 DEBUG:teuthology.exit:Finished running handlers